2016-12-07_01:22:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  running from /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/arvados-cli-0.1.20161017193526/bin/crunch-job with arvados-cli Gem version(s) 0.1.20161017193526, 0.1.20160503204200, 0.1.20151207150126, 0.1.20151023190001
2016-12-07_01:22:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  check slurm allocation
2016-12-07_01:22:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  node humgen-05-15 - 8 slots
2016-12-07_01:22:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  start
2016-12-07_01:22:13 z8ta6-8i9sb-r52unvxj6avgrio 58297  clean work dirs: start
2016-12-07_01:22:13 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','bash','-ec','-o','pipefail','mount -t fuse,fuse.keep | awk "(index(\\$3, \\"$CRUNCH_TMP\\") == 1){print \\$3}" | xargs -r -n 1 fusermount -u -z; sleep 1; rm -rf $JOB_WORK $CRUNCH_INSTALL $CRUNCH_TMP/task $CRUNCH_TMP/src* $CRUNCH_TMP/*.cid']
2016-12-07_01:22:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  clean work dirs: exit 0
2016-12-07_01:22:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  Install docker image 9289c4f1fce1d1dedf9cc7876c33f936+671
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  docker image hash is 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  load docker image: start
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr starting: ['srun','--nodelist=humgen-05-15','/bin/bash','-o','pipefail','-ec',' if /usr/bin/docker images -q --no-trunc --all | grep -xF 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 >/dev/null; then     exit 0 fi declare -a exit_codes=("${PIPESTATUS[@]}") if [ 0 != "${exit_codes[0]}" ]; then    exit "${exit_codes[0]}"  # `docker images` failed elif [ 1 != "${exit_codes[1]}" ]; then    exit "${exit_codes[1]}"  # `grep` encountered an error else    # Everything worked fine, but grep didn\'t find the image on this host.    arv-get 9289c4f1fce1d1dedf9cc7876c33f936\\+671\\/571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3\\.tar | /usr/bin/docker load fi ']
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  load docker image: exit 0
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  check --memory-swap feature: start
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr starting: ['srun','--nodes=1','/usr/bin/docker','run','--help']
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  check --memory-swap feature: exit 0
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  check whether default user is UID 0: start
2016-12-07_01:22:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr starting: ['srun','--nodes=1','/bin/sh','-ec','/usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21  571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 id --user']
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  check whether default user is UID 0: exit 0
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  Container will run with default user
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  Looking for version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 from repository jr17/hgi
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  Using local repository '/var/lib/arvados/internal.git'
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  Version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 is commit 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  run install script on all workers: start
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio','sh','-c','mkdir -p /data/crunch-tmp/crunch-job/opt && cd /data/crunch-tmp/crunch-job && perl -']
2016-12-07_01:22:17 z8ta6-8i9sb-r52unvxj6avgrio 58297  stderr Extracting archive: 4b8cf8e1274f8d418cb494d3ea978152
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  run install script on all workers: exit 0
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  script gatk-combinegvcfs.py
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  script_version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  script_parameters {"interval_lists_collection":"4a72f79fd2ed51ac692bc326fb168f37+10043","reference_collection":"1b447d1aa006cde57df03042736750b9+7894","inputs_collection":"b0dd82bb014b6ab8e64968ebdc07aad9+2566847"}
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  runtime_constraints {"keep_cache_mb_per_task":19500,"max_tasks_per_node":8,"min_nodes":1,"docker_image":"mercury/gatk-3.6"}
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  start level 0 with 1 slots
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 0 done, 0 running, 1 todo
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 job_task z8ta6-ot0gb-ipmynnqm3amv8n7
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 child 58563 started on humgen-05-15.1
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.0.58563','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (1 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ipmynnqm3amv8n7-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ipmynnqm3amv8n7-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ipmynnqm3amv8n7-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=0 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=0 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ipmynnqm3amv8n7 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 0 done, 1 running, 0 todo
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ipmynnqm3amv8n7-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ipmynnqm3amv8n7-0.cid --sig-proxy --memory=250853295k --memory-swap=284407723k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=0 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=0 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ipmynnqm3amv8n7 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8e1a070c037fb957b86cfcd046136e5623d27d4c9885e46b98dc8b52c1229057/cgroup.procs
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8e1a070c037fb957b86cfcd046136e5623d27d4c9885e46b98dc8b52c1229057/memory.stat
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4775936 rss
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8e1a070c037fb957b86cfcd046136e5623d27d4c9885e46b98dc8b52c1229057/cpuacct.stat
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8e1a070c037fb957b86cfcd046136e5623d27d4c9885e46b98dc8b52c1229057/cpuset.cpus
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:22:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8e1a070c037fb957b86cfcd046136e5623d27d4c9885e46b98dc8b52c1229057/blkio.io_service_bytes
2016-12-07_01:22:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:22:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Processing last stream
2016-12-07_01:22:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Finalising stream .
2016-12-07_01:22:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 100_of_200
2016-12-07_01:22:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 100_of_200
2016-12-07_01:22:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 101_of_200
2016-12-07_01:22:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 101_of_200
2016-12-07_01:22:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 102_of_200
2016-12-07_01:22:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 102_of_200
2016-12-07_01:22:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 103_of_200
2016-12-07_01:22:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 103_of_200
2016-12-07_01:22:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 104_of_200
2016-12-07_01:22:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 104_of_200
2016-12-07_01:22:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 105_of_200
2016-12-07_01:22:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 105_of_200
2016-12-07_01:22:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 106_of_200
2016-12-07_01:22:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 106_of_200
2016-12-07_01:22:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 107_of_200
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 107_of_200
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 108_of_200
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 108_of_200
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 109_of_200
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 160366592 rss
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 4.5000 user 0.4300 sys 32 cpus -- interval 10.0001 seconds 4.4700 user 0.4100 sys
2016-12-07_01:22:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 256993 tx 3633431 rx
2016-12-07_01:22:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 109_of_200
2016-12-07_01:22:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 10_of_200
2016-12-07_01:22:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 10_of_200
2016-12-07_01:22:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 110_of_200
2016-12-07_01:22:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 110_of_200
2016-12-07_01:22:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 111_of_200
2016-12-07_01:22:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 111_of_200
2016-12-07_01:22:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 112_of_200
2016-12-07_01:22:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 112_of_200
2016-12-07_01:22:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 113_of_200
2016-12-07_01:22:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 113_of_200
2016-12-07_01:22:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 114_of_200
2016-12-07_01:22:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 114_of_200
2016-12-07_01:22:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 115_of_200
2016-12-07_01:22:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 115_of_200
2016-12-07_01:22:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 116_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 116_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 117_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 117_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 118_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 118_of_200
2016-12-07_01:22:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 119_of_200
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 119_of_200
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 11_of_200
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 11_of_200
2016-12-07_01:22:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 120_of_200
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 173228032 rss
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 6.2200 user 0.4800 sys 32 cpus -- interval 10.0000 seconds 1.7200 user 0.0500 sys
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 545008 tx 4149949 rx -- interval 10.0000 seconds 288015 tx 516518 rx
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 120_of_200
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 121_of_200
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 121_of_200
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 122_of_200
2016-12-07_01:22:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 122_of_200
2016-12-07_01:22:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 123_of_200
2016-12-07_01:22:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 123_of_200
2016-12-07_01:22:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 124_of_200
2016-12-07_01:22:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 124_of_200
2016-12-07_01:22:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 125_of_200
2016-12-07_01:22:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 125_of_200
2016-12-07_01:22:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 126_of_200
2016-12-07_01:22:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 126_of_200
2016-12-07_01:22:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 127_of_200
2016-12-07_01:22:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 127_of_200
2016-12-07_01:22:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 128_of_200
2016-12-07_01:22:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 128_of_200
2016-12-07_01:22:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 129_of_200
2016-12-07_01:22:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 129_of_200
2016-12-07_01:22:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 12_of_200
2016-12-07_01:22:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 12_of_200
2016-12-07_01:22:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 130_of_200
2016-12-07_01:22:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 130_of_200
2016-12-07_01:22:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 131_of_200
2016-12-07_01:22:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 131_of_200
2016-12-07_01:22:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 132_of_200
2016-12-07_01:22:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 132_of_200
2016-12-07_01:22:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 133_of_200
2016-12-07_01:22:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 133_of_200
2016-12-07_01:22:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 134_of_200
2016-12-07_01:22:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 134_of_200
2016-12-07_01:22:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 135_of_200
2016-12-07_01:22:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 135_of_200
2016-12-07_01:22:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 136_of_200
2016-12-07_01:22:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 136_of_200
2016-12-07_01:22:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 137_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 137_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 138_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 138_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 139_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 139_of_200
2016-12-07_01:22:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 13_of_200
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 13_of_200
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 140_of_200
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 140_of_200
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 141_of_200
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 179347456 rss
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 9.4900 user 0.6300 sys 32 cpus -- interval 10.0000 seconds 3.2700 user 0.1500 sys
2016-12-07_01:22:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 1093386 tx 5073141 rx -- interval 10.0000 seconds 548378 tx 923192 rx
2016-12-07_01:22:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 141_of_200
2016-12-07_01:22:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 142_of_200
2016-12-07_01:22:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 142_of_200
2016-12-07_01:22:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 143_of_200
2016-12-07_01:22:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 143_of_200
2016-12-07_01:22:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 144_of_200
2016-12-07_01:22:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 144_of_200
2016-12-07_01:22:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 145_of_200
2016-12-07_01:22:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 145_of_200
2016-12-07_01:22:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 146_of_200
2016-12-07_01:22:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 146_of_200
2016-12-07_01:22:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 147_of_200
2016-12-07_01:22:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 147_of_200
2016-12-07_01:22:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 148_of_200
2016-12-07_01:22:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 148_of_200
2016-12-07_01:22:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 149_of_200
2016-12-07_01:22:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 149_of_200
2016-12-07_01:22:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 14_of_200
2016-12-07_01:22:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 14_of_200
2016-12-07_01:22:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 150_of_200
2016-12-07_01:22:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 150_of_200
2016-12-07_01:22:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 151_of_200
2016-12-07_01:22:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 151_of_200
2016-12-07_01:22:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 152_of_200
2016-12-07_01:22:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 152_of_200
2016-12-07_01:22:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 153_of_200
2016-12-07_01:22:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 153_of_200
2016-12-07_01:22:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 154_of_200
2016-12-07_01:22:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 154_of_200
2016-12-07_01:22:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 155_of_200
2016-12-07_01:22:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 155_of_200
2016-12-07_01:22:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 156_of_200
2016-12-07_01:22:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 156_of_200
2016-12-07_01:22:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 157_of_200
2016-12-07_01:22:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 157_of_200
2016-12-07_01:22:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 158_of_200
2016-12-07_01:22:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 158_of_200
2016-12-07_01:22:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 159_of_200
2016-12-07_01:22:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 159_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 15_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 15_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 160_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 160_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 161_of_200
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 197804032 rss
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 12.4700 user 0.7600 sys 32 cpus -- interval 10.0000 seconds 2.9800 user 0.1300 sys
2016-12-07_01:22:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 1636503 tx 5970742 rx -- interval 10.0000 seconds 543117 tx 897601 rx
2016-12-07_01:22:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 161_of_200
2016-12-07_01:22:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 162_of_200
2016-12-07_01:22:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 162_of_200
2016-12-07_01:22:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 163_of_200
2016-12-07_01:22:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 163_of_200
2016-12-07_01:23:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 164_of_200
2016-12-07_01:23:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 164_of_200
2016-12-07_01:23:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 165_of_200
2016-12-07_01:23:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 165_of_200
2016-12-07_01:23:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 166_of_200
2016-12-07_01:23:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 166_of_200
2016-12-07_01:23:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 167_of_200
2016-12-07_01:23:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 167_of_200
2016-12-07_01:23:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 168_of_200
2016-12-07_01:23:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 168_of_200
2016-12-07_01:23:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 169_of_200
2016-12-07_01:23:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 169_of_200
2016-12-07_01:23:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 16_of_200
2016-12-07_01:23:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 16_of_200
2016-12-07_01:23:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 170_of_200
2016-12-07_01:23:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 170_of_200
2016-12-07_01:23:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 171_of_200
2016-12-07_01:23:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 171_of_200
2016-12-07_01:23:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 172_of_200
2016-12-07_01:23:04 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 172_of_200
2016-12-07_01:23:04 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 173_of_200
2016-12-07_01:23:04 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 173_of_200
2016-12-07_01:23:04 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 174_of_200
2016-12-07_01:23:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 174_of_200
2016-12-07_01:23:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 175_of_200
2016-12-07_01:23:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 175_of_200
2016-12-07_01:23:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 176_of_200
2016-12-07_01:23:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 176_of_200
2016-12-07_01:23:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 177_of_200
2016-12-07_01:23:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 177_of_200
2016-12-07_01:23:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 178_of_200
2016-12-07_01:23:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 178_of_200
2016-12-07_01:23:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 179_of_200
2016-12-07_01:23:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 179_of_200
2016-12-07_01:23:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 17_of_200
2016-12-07_01:23:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 17_of_200
2016-12-07_01:23:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 180_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 180_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 181_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 181_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 182_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 182_of_200
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 226070528 rss
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 15.5300 user 0.9100 sys 32 cpus -- interval 10.0000 seconds 3.0600 user 0.1500 sys
2016-12-07_01:23:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 2183539 tx 6918617 rx -- interval 10.0000 seconds 547036 tx 947875 rx
2016-12-07_01:23:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 183_of_200
2016-12-07_01:23:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 183_of_200
2016-12-07_01:23:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 184_of_200
2016-12-07_01:23:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 184_of_200
2016-12-07_01:23:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 185_of_200
2016-12-07_01:23:10 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 185_of_200
2016-12-07_01:23:10 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 186_of_200
2016-12-07_01:23:10 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 186_of_200
2016-12-07_01:23:10 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 187_of_200
2016-12-07_01:23:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 187_of_200
2016-12-07_01:23:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 188_of_200
2016-12-07_01:23:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 188_of_200
2016-12-07_01:23:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 189_of_200
2016-12-07_01:23:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 189_of_200
2016-12-07_01:23:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 18_of_200
2016-12-07_01:23:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 18_of_200
2016-12-07_01:23:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 190_of_200
2016-12-07_01:23:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 190_of_200
2016-12-07_01:23:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 191_of_200
2016-12-07_01:23:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 191_of_200
2016-12-07_01:23:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 192_of_200
2016-12-07_01:23:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 192_of_200
2016-12-07_01:23:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 193_of_200
2016-12-07_01:23:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 193_of_200
2016-12-07_01:23:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 194_of_200
2016-12-07_01:23:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 194_of_200
2016-12-07_01:23:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 195_of_200
2016-12-07_01:23:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 195_of_200
2016-12-07_01:23:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 196_of_200
2016-12-07_01:23:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 196_of_200
2016-12-07_01:23:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 197_of_200
2016-12-07_01:23:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 197_of_200
2016-12-07_01:23:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 198_of_200
2016-12-07_01:23:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 198_of_200
2016-12-07_01:23:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 199_of_200
2016-12-07_01:23:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 199_of_200
2016-12-07_01:23:17 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 19_of_200
2016-12-07_01:23:17 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 19_of_200
2016-12-07_01:23:17 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 1_of_200
2016-12-07_01:23:17 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 1_of_200
2016-12-07_01:23:17 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 200_of_200
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 200_of_200
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 20_of_200
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 20_of_200
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 21_of_200
2016-12-07_01:23:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 254140416 rss
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 18.5000 user 1.0700 sys 32 cpus -- interval 10.0000 seconds 2.9700 user 0.1600 sys
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 2707575 tx 7815380 rx -- interval 10.0000 seconds 524036 tx 896763 rx
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 21_of_200
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 22_of_200
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 22_of_200
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 23_of_200
2016-12-07_01:23:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 23_of_200
2016-12-07_01:23:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 24_of_200
2016-12-07_01:23:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 24_of_200
2016-12-07_01:23:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 25_of_200
2016-12-07_01:23:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 25_of_200
2016-12-07_01:23:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 26_of_200
2016-12-07_01:23:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 26_of_200
2016-12-07_01:23:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 27_of_200
2016-12-07_01:23:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 27_of_200
2016-12-07_01:23:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 28_of_200
2016-12-07_01:23:22 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 28_of_200
2016-12-07_01:23:22 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 29_of_200
2016-12-07_01:23:22 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 29_of_200
2016-12-07_01:23:22 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 2_of_200
2016-12-07_01:23:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 2_of_200
2016-12-07_01:23:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 30_of_200
2016-12-07_01:23:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 30_of_200
2016-12-07_01:23:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 31_of_200
2016-12-07_01:23:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 31_of_200
2016-12-07_01:23:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 32_of_200
2016-12-07_01:23:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 32_of_200
2016-12-07_01:23:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 33_of_200
2016-12-07_01:23:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 33_of_200
2016-12-07_01:23:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 34_of_200
2016-12-07_01:23:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 34_of_200
2016-12-07_01:23:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 35_of_200
2016-12-07_01:23:25 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 35_of_200
2016-12-07_01:23:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 36_of_200
2016-12-07_01:23:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 36_of_200
2016-12-07_01:23:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 37_of_200
2016-12-07_01:23:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 37_of_200
2016-12-07_01:23:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 38_of_200
2016-12-07_01:23:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 38_of_200
2016-12-07_01:23:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 39_of_200
2016-12-07_01:23:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 39_of_200
2016-12-07_01:23:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 3_of_200
2016-12-07_01:23:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 3_of_200
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 40_of_200
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 40_of_200
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 41_of_200
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 41_of_200
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 280858624 rss
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 21.4900 user 1.2400 sys 32 cpus -- interval 10.0000 seconds 2.9900 user 0.1700 sys
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 3250753 tx 8733455 rx -- interval 10.0000 seconds 543178 tx 918075 rx
2016-12-07_01:23:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 42_of_200
2016-12-07_01:23:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 42_of_200
2016-12-07_01:23:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 43_of_200
2016-12-07_01:23:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 43_of_200
2016-12-07_01:23:29 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 44_of_200
2016-12-07_01:23:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 44_of_200
2016-12-07_01:23:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 45_of_200
2016-12-07_01:23:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 45_of_200
2016-12-07_01:23:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 46_of_200
2016-12-07_01:23:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 46_of_200
2016-12-07_01:23:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 47_of_200
2016-12-07_01:23:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 47_of_200
2016-12-07_01:23:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 48_of_200
2016-12-07_01:23:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 48_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 49_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 49_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 4_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 4_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 50_of_200
2016-12-07_01:23:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 50_of_200
2016-12-07_01:23:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 51_of_200
2016-12-07_01:23:36 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 51_of_200
2016-12-07_01:23:36 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 52_of_200
2016-12-07_01:23:36 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 52_of_200
2016-12-07_01:23:36 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 53_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 53_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 54_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 54_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 55_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 55_of_200
2016-12-07_01:23:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 56_of_200
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 56_of_200
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 57_of_200
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 57_of_200
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 58_of_200
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 302202880 rss
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 23.8300 user 1.3400 sys 32 cpus -- interval 10.0000 seconds 2.3400 user 0.1000 sys
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 3675202 tx 9431114 rx -- interval 10.0000 seconds 424449 tx 697659 rx
2016-12-07_01:23:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 58_of_200
2016-12-07_01:23:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 59_of_200
2016-12-07_01:23:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 59_of_200
2016-12-07_01:23:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 5_of_200
2016-12-07_01:23:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 5_of_200
2016-12-07_01:23:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 60_of_200
2016-12-07_01:23:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 60_of_200
2016-12-07_01:23:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 61_of_200
2016-12-07_01:23:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 61_of_200
2016-12-07_01:23:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 62_of_200
2016-12-07_01:23:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 62_of_200
2016-12-07_01:23:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 63_of_200
2016-12-07_01:23:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 63_of_200
2016-12-07_01:23:41 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 64_of_200
2016-12-07_01:23:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 64_of_200
2016-12-07_01:23:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 65_of_200
2016-12-07_01:23:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 65_of_200
2016-12-07_01:23:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 66_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 66_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 67_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 67_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 68_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 68_of_200
2016-12-07_01:23:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 69_of_200
2016-12-07_01:23:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 69_of_200
2016-12-07_01:23:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 6_of_200
2016-12-07_01:23:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 6_of_200
2016-12-07_01:23:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 70_of_200
2016-12-07_01:23:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 70_of_200
2016-12-07_01:23:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 71_of_200
2016-12-07_01:23:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 71_of_200
2016-12-07_01:23:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 72_of_200
2016-12-07_01:23:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 72_of_200
2016-12-07_01:23:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 73_of_200
2016-12-07_01:23:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 73_of_200
2016-12-07_01:23:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 74_of_200
2016-12-07_01:23:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 74_of_200
2016-12-07_01:23:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 75_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 75_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 76_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 76_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 77_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 77_of_200
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 309436416 rss
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 27.1400 user 1.6200 sys 32 cpus -- interval 10.0000 seconds 3.3100 user 0.2800 sys
2016-12-07_01:23:48 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 4190913 tx 10309781 rx -- interval 10.0000 seconds 515711 tx 878667 rx
2016-12-07_01:23:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 78_of_200
2016-12-07_01:23:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 78_of_200
2016-12-07_01:23:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 79_of_200
2016-12-07_01:23:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 79_of_200
2016-12-07_01:23:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 7_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 7_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 80_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 80_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 81_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 81_of_200
2016-12-07_01:23:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 82_of_200
2016-12-07_01:23:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 82_of_200
2016-12-07_01:23:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 83_of_200
2016-12-07_01:23:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 83_of_200
2016-12-07_01:23:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 84_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 84_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 85_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 85_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 86_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 86_of_200
2016-12-07_01:23:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 87_of_200
2016-12-07_01:23:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 87_of_200
2016-12-07_01:23:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 88_of_200
2016-12-07_01:23:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 88_of_200
2016-12-07_01:23:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 89_of_200
2016-12-07_01:23:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 89_of_200
2016-12-07_01:23:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 8_of_200
2016-12-07_01:23:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 8_of_200
2016-12-07_01:23:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 90_of_200
2016-12-07_01:23:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 90_of_200
2016-12-07_01:23:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 91_of_200
2016-12-07_01:23:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 91_of_200
2016-12-07_01:23:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 92_of_200
2016-12-07_01:23:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 92_of_200
2016-12-07_01:23:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 93_of_200
2016-12-07_01:23:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 93_of_200
2016-12-07_01:23:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 94_of_200
2016-12-07_01:23:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 94_of_200
2016-12-07_01:23:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 95_of_200
2016-12-07_01:23:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 95_of_200
2016-12-07_01:23:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 96_of_200
2016-12-07_01:23:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 96_of_200
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 97_of_200
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 97_of_200
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 98_of_200
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 98_of_200
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: mem 946176 cache 0 swap 0 pgmajfault 309436416 rss
2016-12-07_01:23:58 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: cpu 30.2200 user 1.7400 sys 32 cpus -- interval 10.0000 seconds 3.0800 user 0.1200 sys
2016-12-07_01:23:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr crunchstat: net:eth0 4735428 tx 11234763 rx -- interval 10.0000 seconds 544515 tx 924982 rx
2016-12-07_01:23:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 99_of_200
2016-12-07_01:23:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 99_of_200
2016-12-07_01:23:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Have 42 gVCFs in group 9_of_200
2016-12-07_01:23:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Creating task to process 9_of_200
2016-12-07_01:24:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 stderr Ending task 0 successfully
2016-12-07_01:24:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 child 58563 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:24:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 success in 102 seconds
2016-12-07_01:24:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 0 task output (0 bytes): 
2016-12-07_01:24:00 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:24:10 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 200 new tasks from API server.
2016-12-07_01:24:10 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 1 done, 0 running, 200 todo
2016-12-07_01:24:10 z8ta6-8i9sb-r52unvxj6avgrio 58297  wait for last 0 children to finish
2016-12-07_01:24:10 z8ta6-8i9sb-r52unvxj6avgrio 58297  start level 1 with 8 slots
2016-12-07_01:24:10 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 1 done, 0 running, 200 todo
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 job_task z8ta6-ot0gb-n1ij7gzepmmyrnw
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 child 61926 started on humgen-05-15.1
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 job_task z8ta6-ot0gb-3xe7jgu01d1yiud
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 child 61939 started on humgen-05-15.2
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 job_task z8ta6-ot0gb-q5vot4nnen1ql8m
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 child 61943 started on humgen-05-15.3
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 job_task z8ta6-ot0gb-wc4e7gbasnzlk4n
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 child 61954 started on humgen-05-15.4
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 job_task z8ta6-ot0gb-380otvu8nf013pi
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 child 61959 started on humgen-05-15.5
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 job_task z8ta6-ot0gb-8vdh6ynyoup41n6
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 child 61963 started on humgen-05-15.6
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 job_task z8ta6-ot0gb-i83578e97ax5esa
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 child 61975 started on humgen-05-15.7
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 job_task z8ta6-ot0gb-ab4wzw3us93bwkz
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 child 61979 started on humgen-05-15.8
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.1.61926','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-n1ij7gzepmmyrnw-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-n1ij7gzepmmyrnw-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-n1ij7gzepmmyrnw-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=1 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-n1ij7gzepmmyrnw --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-n1ij7gzepmmyrnw-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-n1ij7gzepmmyrnw-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=1 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-n1ij7gzepmmyrnw --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/413506f275ccb915490816bf5069044c15bd5a735f706672166858d7d7d7104e/cgroup.procs
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/413506f275ccb915490816bf5069044c15bd5a735f706672166858d7d7d7104e/memory.stat
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5513216 rss
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/413506f275ccb915490816bf5069044c15bd5a735f706672166858d7d7d7104e/cpuacct.stat
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/413506f275ccb915490816bf5069044c15bd5a735f706672166858d7d7d7104e/cpuset.cpus
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/413506f275ccb915490816bf5069044c15bd5a735f706672166858d7d7d7104e/blkio.io_service_bytes
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.2.61939','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-3xe7jgu01d1yiud-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-3xe7jgu01d1yiud-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-3xe7jgu01d1yiud-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=2 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-3xe7jgu01d1yiud --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.3.61943','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5vot4nnen1ql8m-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-q5vot4nnen1ql8m-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5vot4nnen1ql8m-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=3 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-q5vot4nnen1ql8m --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.4.61954','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-wc4e7gbasnzlk4n-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-wc4e7gbasnzlk4n-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-wc4e7gbasnzlk4n-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=4 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-wc4e7gbasnzlk4n --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.5.61959','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-380otvu8nf013pi-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-380otvu8nf013pi-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-380otvu8nf013pi-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=5 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-380otvu8nf013pi --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.6.61963','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8vdh6ynyoup41n6-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8vdh6ynyoup41n6-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8vdh6ynyoup41n6-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=6 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-8vdh6ynyoup41n6 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.7.61975','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-i83578e97ax5esa-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-i83578e97ax5esa-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-i83578e97ax5esa-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=7 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-i83578e97ax5esa --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.8.61979','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ab4wzw3us93bwkz-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ab4wzw3us93bwkz-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ab4wzw3us93bwkz-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=8 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ab4wzw3us93bwkz --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 1 done, 8 running, 192 todo
2016-12-07_01:24:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:24:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Mounting task input collection to get interval_list
2016-12-07_01:24:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Interval_List collection mounted at /keep/812316f6c95d8641412912523842bbfc+94
2016-12-07_01:24:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Total chunk length is 15618872
2016-12-07_01:24:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:24:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Have 1 intervals
2016-12-07_01:24:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:24:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:24:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:24:14 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:24:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Have 5400 potential reusable task outputs
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Have 5400 potentially reusable tasks
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Creating task to process interval: [9:6148617-21767488]
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr No reusable JobTask matched key parameters [u'100_of_200', u'a241297995deeab501f9bd99dc533315+14700', '9:6148617-21767488', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 stderr Ending task 1 successfully
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 child 61926 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:24:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 success in 9 seconds
2016-12-07_01:24:21 z8ta6-8i9sb-r52unvxj6avgrio 58297 1 task output (0 bytes): 
2016-12-07_01:24:21 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:24:27 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 2 done, 7 running, 193 todo
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 job_task z8ta6-ot0gb-eeqomjqy20rc5ff
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 child 62498 started on humgen-05-15.1
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.9.62498','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-eeqomjqy20rc5ff-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-eeqomjqy20rc5ff-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-eeqomjqy20rc5ff-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=9 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-eeqomjqy20rc5ff --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-3xe7jgu01d1yiud-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-3xe7jgu01d1yiud-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=2 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-3xe7jgu01d1yiud --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43/cgroup.procs
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43/memory.stat
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: mem 69632 cache 0 swap 0 pgmajfault 6004736 rss
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43/cpuacct.stat
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43/cpuset.cpus
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43/blkio.io_service_bytes
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Mounting task input collection to get interval_list
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Interval_List collection mounted at /keep/6af141a235ccb024f6a32fc12ee98dcf+94
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Total chunk length is 15618872
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Have 1 intervals
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:24:28 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 2 done, 8 running, 192 todo
2016-12-07_01:24:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Have 5400 potential reusable task outputs
2016-12-07_01:24:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Have 5400 potentially reusable tasks
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Creating task to process interval: [9:21767489-37386360]
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr No reusable JobTask matched key parameters [u'101_of_200', u'1b848222e216cbae87e6932d5820a285+14700', '9:21767489-37386360', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr Ending task 1 successfully
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: stats not available: stat memory.stat, statgroup memory, cid 52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43, parent docker, root /sys/fs/cgroup
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: stats not available: stat cpuacct.stat, statgroup cpuacct, cid 52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43, parent docker, root /sys/fs/cgroup
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: stats not available: stat blkio.io_service_bytes, statgroup blkio, cid 52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43, parent docker, root /sys/fs/cgroup
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 stderr crunchstat: notice: stats not available: stat cgroup.procs, statgroup cpuacct, cid 52f5ec6c847f37448e83c71573c5b10b9cffdc7398a63b0bbd6132806781bf43, parent docker, root /sys/fs/cgroup
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr srun: Job step created
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 child 61939 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 success in 20 seconds
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 2 task output (0 bytes): 
2016-12-07_01:24:31 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 3 done, 7 running, 193 todo
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 job_task z8ta6-ot0gb-73kg9lkqq5037a4
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 child 62816 started on humgen-05-15.2
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.10.62816','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-73kg9lkqq5037a4-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-73kg9lkqq5037a4-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-73kg9lkqq5037a4-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=10 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-73kg9lkqq5037a4 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-q5vot4nnen1ql8m-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5vot4nnen1ql8m-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=3 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-q5vot4nnen1ql8m --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/dc31b02e7d767e7ad348c942bcf5d554a7bdef046b6bbb387e09e38ffeaafe5a/cgroup.procs
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/dc31b02e7d767e7ad348c942bcf5d554a7bdef046b6bbb387e09e38ffeaafe5a/memory.stat
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 3395584 rss
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/dc31b02e7d767e7ad348c942bcf5d554a7bdef046b6bbb387e09e38ffeaafe5a/cpuacct.stat
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/dc31b02e7d767e7ad348c942bcf5d554a7bdef046b6bbb387e09e38ffeaafe5a/cpuset.cpus
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: cpu 0.0200 user 0.0100 sys 32 cpus
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/dc31b02e7d767e7ad348c942bcf5d554a7bdef046b6bbb387e09e38ffeaafe5a/blkio.io_service_bytes
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Mounting task input collection to get interval_list
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Interval_List collection mounted at /keep/2e7cf8719961c4cc5b2ffdd92ff99dd0+94
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Total chunk length is 15618872
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:24:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Have 1 intervals
2016-12-07_01:24:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:24:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:24:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:24:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:24:39 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 3 done, 8 running, 192 todo
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: cpu 0.5200 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.5000 user 0.1500 sys
2016-12-07_01:24:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr crunchstat: net:eth0 21985 tx 238258 rx
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Have 5400 potential reusable task outputs
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Have 5400 potentially reusable tasks
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Creating task to process interval: [9:37386361-53005232]
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr No reusable JobTask matched key parameters [u'102_of_200', u'0b01f4aa638023c7844e14aa840fe4cc+14532', '9:37386361-53005232', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:24:43 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 stderr Ending task 1 successfully
2016-12-07_01:24:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 child 61943 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:24:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 success in 33 seconds
2016-12-07_01:24:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 3 task output (0 bytes): 
2016-12-07_01:24:44 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 4 done, 7 running, 193 todo
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 job_task z8ta6-ot0gb-q5ob5ca6mm8v0v8
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 child 63162 started on humgen-05-15.3
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.11.63162','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5ob5ca6mm8v0v8-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-q5ob5ca6mm8v0v8-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5ob5ca6mm8v0v8-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=11 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-q5ob5ca6mm8v0v8 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr srun: Job step created
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-380otvu8nf013pi-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-380otvu8nf013pi-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=5 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-380otvu8nf013pi --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/2da3f5be2ed83479116a5ed081e74951462b46e086e97129df9cd80578cf181e/cgroup.procs
2016-12-07_01:24:50 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/2da3f5be2ed83479116a5ed081e74951462b46e086e97129df9cd80578cf181e/memory.stat
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4403200 rss
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/2da3f5be2ed83479116a5ed081e74951462b46e086e97129df9cd80578cf181e/cpuacct.stat
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/2da3f5be2ed83479116a5ed081e74951462b46e086e97129df9cd80578cf181e/cpuset.cpus
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/2da3f5be2ed83479116a5ed081e74951462b46e086e97129df9cd80578cf181e/blkio.io_service_bytes
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Mounting task input collection to get interval_list
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Interval_List collection mounted at /keep/61144aa3079bc748eb474251bb12d2af+94
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Total chunk length is 15618872
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Have 1 intervals
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:24:51 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 4 done, 8 running, 192 todo
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Have 5400 potential reusable task outputs
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Have 5400 potentially reusable tasks
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Creating task to process interval: [9:68624105-84242976]
2016-12-07_01:24:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr No reusable JobTask matched key parameters [u'104_of_200', u'0e7a8a6d12ed4d9a7f7a47c972f98c08+14700', '9:68624105-84242976', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:24:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 stderr Ending task 1 successfully
2016-12-07_01:24:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 child 61959 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:24:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 success in 43 seconds
2016-12-07_01:24:54 z8ta6-8i9sb-r52unvxj6avgrio 58297 5 task output (0 bytes): 
2016-12-07_01:24:54 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 5 done, 7 running, 193 todo
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 job_task z8ta6-ot0gb-kbnkrhgu14765w0
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 child 63553 started on humgen-05-15.5
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr srun: Job step created
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-eeqomjqy20rc5ff-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-eeqomjqy20rc5ff-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=9 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-eeqomjqy20rc5ff --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/10d014edf6d222438b913ff72cf0a31700ba01872e31cf2931ca59ac9b5b7b7c/cgroup.procs
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/10d014edf6d222438b913ff72cf0a31700ba01872e31cf2931ca59ac9b5b7b7c/memory.stat
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7778304 rss
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/10d014edf6d222438b913ff72cf0a31700ba01872e31cf2931ca59ac9b5b7b7c/cpuacct.stat
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/10d014edf6d222438b913ff72cf0a31700ba01872e31cf2931ca59ac9b5b7b7c/cpuset.cpus
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: cpu 0.0100 user 0.0000 sys 32 cpus
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/10d014edf6d222438b913ff72cf0a31700ba01872e31cf2931ca59ac9b5b7b7c/blkio.io_service_bytes
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Mounting task input collection to get interval_list
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Interval_List collection mounted at /keep/64696f9b403b7674c728386882cbd3b7+94
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Total chunk length is 15498872
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Have 1 intervals
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.12.63553','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-kbnkrhgu14765w0-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-kbnkrhgu14765w0-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-kbnkrhgu14765w0-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=12 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-kbnkrhgu14765w0 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:25:01 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 5 done, 8 running, 192 todo
2016-12-07_01:25:04 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: blkio:0:0 0 write 12564 read -- interval 10.0000 seconds 0 write 12564 read
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Have 5400 potential reusable task outputs
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 46039040 rss
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: cpu 0.6600 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.6500 user 0.1800 sys
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr crunchstat: net:eth0 34851 tx 3237537 rx
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Have 5400 potentially reusable tasks
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Creating task to process interval: [9:131099593-141213431 10:1-5385033]
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr No reusable JobTask matched key parameters [u'108_of_200', u'7e64dee294f55ab4bfb56c1a154618f8+14704', '9:131099593-141213431 10:1-5385033', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 stderr Ending task 1 successfully
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 child 62498 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:25:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 success in 37 seconds
2016-12-07_01:25:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 9 task output (0 bytes): 
2016-12-07_01:25:06 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 6 done, 7 running, 193 todo
2016-12-07_01:25:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 job_task z8ta6-ot0gb-0r2ariip11y1uwg
2016-12-07_01:25:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 child 64183 started on humgen-05-15.1
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.13.64183','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0r2ariip11y1uwg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0r2ariip11y1uwg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0r2ariip11y1uwg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=13 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-0r2ariip11y1uwg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr srun: Job step created
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8vdh6ynyoup41n6-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8vdh6ynyoup41n6-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=6 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-8vdh6ynyoup41n6 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e/cgroup.procs
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e/memory.stat
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5545984 rss
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e/cpuacct.stat
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e/cpuset.cpus
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: cpu 0.0400 user 0.0200 sys 32 cpus
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e/blkio.io_service_bytes
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Mounting task input collection to get interval_list
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Interval_List collection mounted at /keep/4325c7dac5c39f2db578e849695a12aa+94
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Total chunk length is 15618872
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Have 1 intervals
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:25:13 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 6 done, 8 running, 192 todo
2016-12-07_01:25:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Have 5400 potential reusable task outputs
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 notice: task is not in slurm queue but srun process 61979 has not exited
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 notice: task is not in slurm queue but srun process 61975 has not exited
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 notice: task is not in slurm queue but srun process 61954 has not exited
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Have 5400 potentially reusable tasks
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Creating task to process interval: [9:84242977-99861848]
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr No reusable JobTask matched key parameters [u'105_of_200', u'3a3bb24e84ba30a06b3d46caa3b1c040+14700', '9:84242977-99861848', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr Ending task 1 successfully
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: stats not available: stat memory.stat, statgroup memory, cid 7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: stats not available: stat cpuacct.stat, statgroup cpuacct, cid 7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: stats not available: stat blkio.io_service_bytes, statgroup blkio, cid 7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 stderr crunchstat: notice: stats not available: stat cgroup.procs, statgroup cpuacct, cid 7a6ab25b3585930bddd4f343598c3b735a14c334c161c02bcae037b955d6716e, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr srun: Job step created
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 child 61963 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 success in 65 seconds
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297 6 task output (0 bytes): 
2016-12-07_01:25:16 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 7 done, 7 running, 193 todo
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 job_task z8ta6-ot0gb-b9qjnxrsbhkkm8l
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 child 64510 started on humgen-05-15.6
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-q5ob5ca6mm8v0v8-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-q5ob5ca6mm8v0v8-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=11 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-q5ob5ca6mm8v0v8 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54/cgroup.procs
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54/memory.stat
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7946240 rss
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54/cpuacct.stat
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54/cpuset.cpus
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54/blkio.io_service_bytes
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Mounting task input collection to get interval_list
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Interval_List collection mounted at /keep/47924fc036d6a931b5f0cf53440e0969+92
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Total chunk length is 15618872
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Have 1 intervals
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:25:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.14.64510','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-b9qjnxrsbhkkm8l-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-b9qjnxrsbhkkm8l-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-b9qjnxrsbhkkm8l-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=14 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-b9qjnxrsbhkkm8l --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:25:24 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 7 done, 8 running, 192 todo
2016-12-07_01:25:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Have 5400 potential reusable task outputs
2016-12-07_01:25:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Have 5400 potentially reusable tasks
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Creating task to process interval: [1:140569849-156188720]
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr No reusable JobTask matched key parameters [u'10_of_200', u'a99a25ff7de535672c0817451b92bd1a+14616', '1:140569849-156188720', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr Ending task 1 successfully
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: stats not available: stat memory.stat, statgroup memory, cid d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: stats not available: stat cpuacct.stat, statgroup cpuacct, cid d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: stats not available: stat blkio.io_service_bytes, statgroup blkio, cid d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 stderr crunchstat: notice: stats not available: stat cgroup.procs, statgroup cpuacct, cid d783d74b19d58c676f1cfdc4ebfabd63e9c2b3e518d86520dad615aa257b3f54, parent docker, root /sys/fs/cgroup
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 child 63162 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 success in 37 seconds
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 11 task output (0 bytes): 
2016-12-07_01:25:27 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 8 done, 7 running, 193 todo
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 job_task z8ta6-ot0gb-k3exaksrtpevoki
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 child 64881 started on humgen-05-15.3
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.15.64881','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-k3exaksrtpevoki-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-k3exaksrtpevoki-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-k3exaksrtpevoki-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=15 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-k3exaksrtpevoki --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr srun: Job step created
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-i83578e97ax5esa-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-i83578e97ax5esa-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=7 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-i83578e97ax5esa --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/08ac59a53360e54c20c24592b99dbf66e94c153030a4d14349856f8be6bed077/cgroup.procs
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/08ac59a53360e54c20c24592b99dbf66e94c153030a4d14349856f8be6bed077/memory.stat
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: mem 16384 cache 0 swap 0 pgmajfault 5697536 rss
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/08ac59a53360e54c20c24592b99dbf66e94c153030a4d14349856f8be6bed077/cpuacct.stat
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/08ac59a53360e54c20c24592b99dbf66e94c153030a4d14349856f8be6bed077/cpuset.cpus
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: cpu 0.0400 user 0.0200 sys 32 cpus
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/08ac59a53360e54c20c24592b99dbf66e94c153030a4d14349856f8be6bed077/blkio.io_service_bytes
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: net:eth0 0 tx 0 rx
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Mounting task input collection to get interval_list
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Interval_List collection mounted at /keep/34f0119dab554faf0acfc04f3cae7c1d+94
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Total chunk length is 15618872
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Have 1 intervals
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:25:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:25:35 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 8 done, 8 running, 192 todo
2016-12-07_01:25:37 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24121344 rss
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: cpu 0.5000 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.4600 user 0.1600 sys
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr crunchstat: net:eth0 22009 tx 238347 rx -- interval 10.0002 seconds 22009 tx 238347 rx
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Have 5400 potential reusable task outputs
2016-12-07_01:25:38 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Have 5400 potentially reusable tasks
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Creating task to process interval: [9:99861849-115480720]
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr No reusable JobTask matched key parameters [u'106_of_200', u'dc8fa207ac10a02e00e1ba8ab1687ad9+14700', '9:99861849-115480720', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 stderr Ending task 1 successfully
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr srun: Job step created
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 child 61975 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 success in 88 seconds
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297 7 task output (0 bytes): 
2016-12-07_01:25:39 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 9 done, 7 running, 193 todo
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 job_task z8ta6-ot0gb-204sempjbouqi2x
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 child 65269 started on humgen-05-15.7
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0r2ariip11y1uwg-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0r2ariip11y1uwg-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=13 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-0r2ariip11y1uwg --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/518065a3ee33c8204bb70183d34f7927ac45961f43200f452eb520231d004b60/cgroup.procs
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/518065a3ee33c8204bb70183d34f7927ac45961f43200f452eb520231d004b60/memory.stat
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4087808 rss
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/518065a3ee33c8204bb70183d34f7927ac45961f43200f452eb520231d004b60/cpuacct.stat
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/518065a3ee33c8204bb70183d34f7927ac45961f43200f452eb520231d004b60/cpuset.cpus
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/518065a3ee33c8204bb70183d34f7927ac45961f43200f452eb520231d004b60/blkio.io_service_bytes
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Mounting task input collection to get interval_list
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Interval_List collection mounted at /keep/d6cf1afdecfe2ae2261292a571a3759b+94
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Total chunk length is 15618872
2016-12-07_01:25:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Have 1 intervals
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.16.65269','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-204sempjbouqi2x-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-204sempjbouqi2x-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-204sempjbouqi2x-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=16 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-204sempjbouqi2x --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:25:47 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 9 done, 8 running, 192 todo
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 killing orphaned srun process 61979 (task not in slurm queue, no stderr received in last 89s)
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 sending 2x signal 2 to pid 61979
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 killing orphaned srun process 61954 (task not in slurm queue, no stderr received in last 89s)
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 sending 2x signal 2 to pid 61954
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 notice: task is not in slurm queue but srun process 62816 has not exited
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 child 61954 on humgen-05-15.4 exit 1 success=
2016-12-07_01:25:49 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:25:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 failure (#1, temporary) after 98 seconds
2016-12-07_01:25:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 4 task output (0 bytes): 
2016-12-07_01:25:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 child 61979 on humgen-05-15.8 exit 1 success=
2016-12-07_01:25:51 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 failure (#1, temporary) after 100 seconds
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 8 task output (0 bytes): 
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 child 64183 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 success in 41 seconds
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 13 task output (0 bytes): 
2016-12-07_01:25:53 z8ta6-8i9sb-r52unvxj6avgrio 58297  3 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:25:59 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:25:59 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 10 done, 5 running, 195 todo
2016-12-07_01:25:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 job_task z8ta6-ot0gb-ttjug7tm91r0b32
2016-12-07_01:25:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 child 465 started on humgen-05-15.4
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 job_task z8ta6-ot0gb-1fbv3vnkrx4i2ix
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 child 473 started on humgen-05-15.8
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 job_task z8ta6-ot0gb-5n5f89qjphfwamo
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 child 477 started on humgen-05-15.1
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.17.465','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ttjug7tm91r0b32-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ttjug7tm91r0b32-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ttjug7tm91r0b32-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=17 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ttjug7tm91r0b32 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.18.473','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1fbv3vnkrx4i2ix-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-1fbv3vnkrx4i2ix-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1fbv3vnkrx4i2ix-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=18 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-1fbv3vnkrx4i2ix --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr srun: Job step created
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-kbnkrhgu14765w0-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-kbnkrhgu14765w0-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=12 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-kbnkrhgu14765w0 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/baf01763ceb31c32caaf390f55aeef36bc024a43c6f63965c7b05d1324ec486f/cgroup.procs
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/baf01763ceb31c32caaf390f55aeef36bc024a43c6f63965c7b05d1324ec486f/memory.stat
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7962624 rss
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/baf01763ceb31c32caaf390f55aeef36bc024a43c6f63965c7b05d1324ec486f/cpuacct.stat
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/baf01763ceb31c32caaf390f55aeef36bc024a43c6f63965c7b05d1324ec486f/cpuset.cpus
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: cpu 0.0200 user 0.0100 sys 32 cpus
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/baf01763ceb31c32caaf390f55aeef36bc024a43c6f63965c7b05d1324ec486f/blkio.io_service_bytes
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Interval_List collection mounted at /keep/11837032d32bcfff7055a484f9c0ddb8+94
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Total chunk length is 15618872
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Have 1 intervals
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.19.477','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5n5f89qjphfwamo-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-5n5f89qjphfwamo-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5n5f89qjphfwamo-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=19 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-5n5f89qjphfwamo --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:00 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 10 done, 8 running, 192 todo
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Have 5400 potentially reusable tasks
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Creating task to process interval: [10:21003906-36622777]
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr No reusable JobTask matched key parameters [u'110_of_200', u'e20f5f00b25cd7e53f799770dbe9e308+14700', '10:21003906-36622777', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:26:01 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 stderr Ending task 1 successfully
2016-12-07_01:26:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr srun: Job step created
2016-12-07_01:26:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 child 63553 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:26:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 success in 61 seconds
2016-12-07_01:26:02 z8ta6-8i9sb-r52unvxj6avgrio 58297 12 task output (0 bytes): 
2016-12-07_01:26:02 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 11 done, 7 running, 193 todo
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 job_task z8ta6-ot0gb-xiz6ocqs0kyyqzj
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 child 796 started on humgen-05-15.5
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.20.796','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xiz6ocqs0kyyqzj-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xiz6ocqs0kyyqzj-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xiz6ocqs0kyyqzj-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=20 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-xiz6ocqs0kyyqzj --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-204sempjbouqi2x-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-204sempjbouqi2x-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=16 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-204sempjbouqi2x --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/3ddcbaebdca3c84192e03ef232a15e0b7c04d923565e5a5ad90f6d630579361b/cgroup.procs
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/3ddcbaebdca3c84192e03ef232a15e0b7c04d923565e5a5ad90f6d630579361b/memory.stat
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: mem 94208 cache 0 swap 0 pgmajfault 5992448 rss
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/3ddcbaebdca3c84192e03ef232a15e0b7c04d923565e5a5ad90f6d630579361b/cpuacct.stat
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/3ddcbaebdca3c84192e03ef232a15e0b7c04d923565e5a5ad90f6d630579361b/cpuset.cpus
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/3ddcbaebdca3c84192e03ef232a15e0b7c04d923565e5a5ad90f6d630579361b/blkio.io_service_bytes
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Interval_List collection mounted at /keep/8b0a6de6be0a7b3dd7360336477e02fd+94
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Total chunk length is 15618872
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Have 1 intervals
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:09 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 11 done, 8 running, 192 todo
2016-12-07_01:26:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:11 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Have 5400 potentially reusable tasks
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Creating task to process interval: [10:83479394-99098265]
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr No reusable JobTask matched key parameters [u'114_of_200', u'c3bc3b07edcf5d2b7b9bfb2b5154d56f+14700', '10:83479394-99098265', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 stderr Ending task 1 successfully
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 child 65269 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 success in 26 seconds
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297 16 task output (0 bytes): 
2016-12-07_01:26:12 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 12 done, 7 running, 193 todo
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 job_task z8ta6-ot0gb-2vh6dr8bmvpcz1x
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 child 1139 started on humgen-05-15.7
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ttjug7tm91r0b32-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ttjug7tm91r0b32-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=17 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ttjug7tm91r0b32 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/79f45e237394f23d11b250d77a9e4f2cefbdba5624d9dd67ac8ce406e2863237/cgroup.procs
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/79f45e237394f23d11b250d77a9e4f2cefbdba5624d9dd67ac8ce406e2863237/memory.stat
2016-12-07_01:26:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5664768 rss
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/79f45e237394f23d11b250d77a9e4f2cefbdba5624d9dd67ac8ce406e2863237/cpuacct.stat
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/79f45e237394f23d11b250d77a9e4f2cefbdba5624d9dd67ac8ce406e2863237/cpuset.cpus
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/79f45e237394f23d11b250d77a9e4f2cefbdba5624d9dd67ac8ce406e2863237/blkio.io_service_bytes
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Interval_List collection mounted at /keep/e05c9a56f87bbb0e87bd78edfb2138bb+94
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Total chunk length is 15618872
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Have 1 intervals
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.21.1139','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2vh6dr8bmvpcz1x-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2vh6dr8bmvpcz1x-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2vh6dr8bmvpcz1x-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=21 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-2vh6dr8bmvpcz1x --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:20 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 12 done, 8 running, 192 todo
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: blkio:0:0 0 write 12527 read -- interval 10.0000 seconds 0 write 12527 read
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24084480 rss
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: cpu 0.5000 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.4600 user 0.1500 sys
2016-12-07_01:26:23 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr crunchstat: net:eth0 22273 tx 238495 rx
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 killing orphaned srun process 62816 (task not in slurm queue, no stderr received in last 94s)
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 sending 2x signal 2 to pid 62816
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 notice: task is not in slurm queue but srun process 64510 has not exited
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 child 62816 on humgen-05-15.2 exit 1 success=
2016-12-07_01:26:24 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 failure (#1, temporary) after 106 seconds
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 10 task output (0 bytes): 
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 child 465 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 success in 27 seconds
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297 17 task output (0 bytes): 
2016-12-07_01:26:26 z8ta6-8i9sb-r52unvxj6avgrio 58297  2 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:26:32 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 13 done, 6 running, 194 todo
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 job_task z8ta6-ot0gb-8zpggr6kb52w373
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 child 1517 started on humgen-05-15.2
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 job_task z8ta6-ot0gb-1uoyxunyt78l0t9
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 child 1522 started on humgen-05-15.4
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.22.1517','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8zpggr6kb52w373-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8zpggr6kb52w373-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8zpggr6kb52w373-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=22 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-8zpggr6kb52w373 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.23.1522','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1uoyxunyt78l0t9-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-1uoyxunyt78l0t9-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1uoyxunyt78l0t9-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=23 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-1uoyxunyt78l0t9 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xiz6ocqs0kyyqzj-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xiz6ocqs0kyyqzj-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=20 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-xiz6ocqs0kyyqzj --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/78a3e96cdfe2fd39b7e45635dd2596382e56dfcde671fe140fb12be811c87708/cgroup.procs
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/78a3e96cdfe2fd39b7e45635dd2596382e56dfcde671fe140fb12be811c87708/memory.stat
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: mem 94208 cache 0 swap 0 pgmajfault 6393856 rss
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/78a3e96cdfe2fd39b7e45635dd2596382e56dfcde671fe140fb12be811c87708/cpuacct.stat
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/78a3e96cdfe2fd39b7e45635dd2596382e56dfcde671fe140fb12be811c87708/cpuset.cpus
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: cpu 0.0400 user 0.0200 sys 32 cpus
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/78a3e96cdfe2fd39b7e45635dd2596382e56dfcde671fe140fb12be811c87708/blkio.io_service_bytes
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Interval_List collection mounted at /keep/d9aa702cd7b3a1e6b6dec54b4a3d8e35+94
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Total chunk length is 15618872
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Have 1 intervals
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:33 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 13 done, 8 running, 192 todo
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Have 5400 potentially reusable tasks
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Creating task to process interval: [11:10300135-25919006]
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr No reusable JobTask matched key parameters [u'118_of_200', u'79b22e2a5c2a107fd090c78ed23ff2e7+14700', '11:10300135-25919006', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:26:34 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 stderr Ending task 1 successfully
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr srun: Job step created
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 child 796 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 success in 26 seconds
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297 20 task output (0 bytes): 
2016-12-07_01:26:35 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 14 done, 7 running, 193 todo
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 job_task z8ta6-ot0gb-2r646fu5fbu5irj
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 child 1854 started on humgen-05-15.5
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-k3exaksrtpevoki-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-k3exaksrtpevoki-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=15 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-k3exaksrtpevoki --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a30eab30e6e495073442c82d6f54535f9cb88e32e35a2610bea4e2302aac689b/cgroup.procs
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/a30eab30e6e495073442c82d6f54535f9cb88e32e35a2610bea4e2302aac689b/memory.stat
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 884736 rss
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a30eab30e6e495073442c82d6f54535f9cb88e32e35a2610bea4e2302aac689b/cpuacct.stat
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/a30eab30e6e495073442c82d6f54535f9cb88e32e35a2610bea4e2302aac689b/cpuset.cpus
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: cpu 0.0100 user 0.0000 sys 32 cpus
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/a30eab30e6e495073442c82d6f54535f9cb88e32e35a2610bea4e2302aac689b/blkio.io_service_bytes
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Interval_List collection mounted at /keep/f73eab8bfca061b956c4cbaa59d62ccd+94
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Total chunk length is 15618872
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Have 1 intervals
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.24.1854','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2r646fu5fbu5irj-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2r646fu5fbu5irj-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2r646fu5fbu5irj-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=24 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-2r646fu5fbu5irj --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:42 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 14 done, 8 running, 192 todo
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Have 5400 potentially reusable tasks
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Creating task to process interval: [10:67860522-83479393]
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr No reusable JobTask matched key parameters [u'113_of_200', u'45d9d4c732c4ecae90a7f5ec11993512+14700', '10:67860522-83479393', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr Ending task 1 successfully
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:26:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:26:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 child 64881 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:26:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 success in 72 seconds
2016-12-07_01:26:46 z8ta6-8i9sb-r52unvxj6avgrio 58297 15 task output (0 bytes): 
2016-12-07_01:26:46 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:26:52 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 15 done, 7 running, 193 todo
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 job_task z8ta6-ot0gb-h0sz2rle16hsfng
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 child 2495 started on humgen-05-15.3
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-1uoyxunyt78l0t9-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1uoyxunyt78l0t9-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=23 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-1uoyxunyt78l0t9 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/9f713a5e029867565cc57b51270f1925b83d8c2ac780c56bae24d6d16ad63651/cgroup.procs
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/9f713a5e029867565cc57b51270f1925b83d8c2ac780c56bae24d6d16ad63651/memory.stat
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5427200 rss
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/9f713a5e029867565cc57b51270f1925b83d8c2ac780c56bae24d6d16ad63651/cpuacct.stat
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/9f713a5e029867565cc57b51270f1925b83d8c2ac780c56bae24d6d16ad63651/cpuset.cpus
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/9f713a5e029867565cc57b51270f1925b83d8c2ac780c56bae24d6d16ad63651/blkio.io_service_bytes
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Mounting task input collection to get interval_list
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Interval_List collection mounted at /keep/b34c79e916b3f4868c677e831b11c26d+94
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Total chunk length is 15618872
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Have 1 intervals
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.25.2495','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h0sz2rle16hsfng-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-h0sz2rle16hsfng-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h0sz2rle16hsfng-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=25 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-h0sz2rle16hsfng --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:26:53 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 15 done, 8 running, 192 todo
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: cpu 0.4700 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.4400 user 0.1700 sys
2016-12-07_01:26:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr crunchstat: net:eth0 21853 tx 238207 rx
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 killing orphaned srun process 64510 (task not in slurm queue, no stderr received in last 83s)
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 sending 2x signal 2 to pid 64510
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Have 5400 potential reusable task outputs
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 child 64510 on humgen-05-15.6 exit 1 success=
2016-12-07_01:26:57 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 failure (#1, temporary) after 94 seconds
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 14 task output (0 bytes): 
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 child 1522 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 success in 26 seconds
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297 23 task output (0 bytes): 
2016-12-07_01:26:59 z8ta6-8i9sb-r52unvxj6avgrio 58297  2 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 16 done, 6 running, 194 todo
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297  Every node has failed -- giving up
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297  wait for last 6 children to finish
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr srun: Job step created
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-1fbv3vnkrx4i2ix-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-1fbv3vnkrx4i2ix-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=18 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-1fbv3vnkrx4i2ix --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/5384010f071eca3184f862a81e2d766f90f105041091865336b4254df938643b/cgroup.procs
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/5384010f071eca3184f862a81e2d766f90f105041091865336b4254df938643b/memory.stat
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4861952 rss
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/5384010f071eca3184f862a81e2d766f90f105041091865336b4254df938643b/cpuacct.stat
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/5384010f071eca3184f862a81e2d766f90f105041091865336b4254df938643b/cpuset.cpus
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/5384010f071eca3184f862a81e2d766f90f105041091865336b4254df938643b/blkio.io_service_bytes
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:27:05 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Mounting task input collection to get interval_list
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Interval_List collection mounted at /keep/359a96b521b56e8d8e36b57450fa80ac+94
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Total chunk length is 15618872
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Have 1 intervals
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:27:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:27:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Have 5400 potential reusable task outputs
2016-12-07_01:27:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:27:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Have 5400 potentially reusable tasks
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Creating task to process interval: [10:114717138-130336009]
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr No reusable JobTask matched key parameters [u'116_of_200', u'6365525c0ee094fee362794dbbab6a30+14700', '10:114717138-130336009', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr Ending task 1 successfully
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 stderr crunchstat: blkio:0:0 0 write 12528 read -- interval 10.0000 seconds 0 write 12528 read
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 child 473 on humgen-05-15.8 exit 0 success=true
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 success in 68 seconds
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 18 task output (0 bytes): 
2016-12-07_01:27:08 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-h0sz2rle16hsfng-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-h0sz2rle16hsfng-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=25 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-h0sz2rle16hsfng --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/18ff3a339d9f9420eef93cc70e123155780365d31f61a629d7c1d63e749a8b5f/cgroup.procs
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/18ff3a339d9f9420eef93cc70e123155780365d31f61a629d7c1d63e749a8b5f/memory.stat
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 1810432 rss
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/18ff3a339d9f9420eef93cc70e123155780365d31f61a629d7c1d63e749a8b5f/cpuacct.stat
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/18ff3a339d9f9420eef93cc70e123155780365d31f61a629d7c1d63e749a8b5f/cpuset.cpus
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: cpu 0.0200 user 0.0200 sys 32 cpus
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/18ff3a339d9f9420eef93cc70e123155780365d31f61a629d7c1d63e749a8b5f/blkio.io_service_bytes
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Mounting task input collection to get interval_list
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Interval_List collection mounted at /keep/bbfdffd38f1432cef552ced110ab3a90+94
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Total chunk length is 15618872
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Have 1 intervals
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 notice: task is not in slurm queue but srun process 477 has not exited
2016-12-07_01:27:15 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 17 done, 5 running, 195 todo
2016-12-07_01:27:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:27:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:27:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:27:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:27:18 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:27:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:27:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: cpu 0.5000 user 0.1800 sys 32 cpus -- interval 10.0001 seconds 0.4800 user 0.1600 sys
2016-12-07_01:27:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr crunchstat: net:eth0 21919 tx 238193 rx
2016-12-07_01:27:19 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Have 5400 potential reusable task outputs
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Have 5400 potentially reusable tasks
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Creating task to process interval: [11:72775623-88394494]
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr No reusable JobTask matched key parameters [u'122_of_200', u'af62e8edc576e4baf02413c2134c9270+14700', '11:72775623-88394494', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 stderr Ending task 1 successfully
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 child 2495 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 success in 27 seconds
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297 25 task output (0 bytes): 
2016-12-07_01:27:20 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr srun: Job step created
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-5n5f89qjphfwamo-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5n5f89qjphfwamo-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=19 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-5n5f89qjphfwamo --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/f0c64858772c6b586dffaff4b556f34c458300b982d1c421ce9a68178f633a32/cgroup.procs
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/f0c64858772c6b586dffaff4b556f34c458300b982d1c421ce9a68178f633a32/memory.stat
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 258048 rss
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/f0c64858772c6b586dffaff4b556f34c458300b982d1c421ce9a68178f633a32/cpuacct.stat
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/f0c64858772c6b586dffaff4b556f34c458300b982d1c421ce9a68178f633a32/cpuset.cpus
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: cpu 0.0100 user 0.0100 sys 32 cpus
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/f0c64858772c6b586dffaff4b556f34c458300b982d1c421ce9a68178f633a32/blkio.io_service_bytes
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Mounting task input collection to get interval_list
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Interval_List collection mounted at /keep/76a230c91cc5d0053b75306d58906b2a+94
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Total chunk length is 15498872
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Have 1 intervals
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:27:27 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 18 done, 4 running, 196 todo
2016-12-07_01:27:30 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 notice: task is not in slurm queue but srun process 1139 has not exited
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: blkio:0:0 0 write 12567 read -- interval 10.0000 seconds 0 write 12567 read
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24084480 rss
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: cpu 0.5200 user 0.1500 sys 32 cpus -- interval 10.0001 seconds 0.5100 user 0.1400 sys
2016-12-07_01:27:31 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr crunchstat: net:eth0 21807 tx 238026 rx
2016-12-07_01:27:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Have 5400 potential reusable task outputs
2016-12-07_01:27:32 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Have 5400 potentially reusable tasks
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Creating task to process interval: [10:130336010-135534747 11:1-10300134]
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr No reusable JobTask matched key parameters [u'117_of_200', u'a986cd76d508917b0b96bd60c6107961+14704', '10:130336010-135534747 11:1-10300134', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 stderr Ending task 1 successfully
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr srun: Job step created
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 child 477 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 success in 93 seconds
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297 19 task output (0 bytes): 
2016-12-07_01:27:33 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2r646fu5fbu5irj-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2r646fu5fbu5irj-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=24 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-2r646fu5fbu5irj --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d7575a9a863245702665a02fc3af5f784c19382aac8b880fe4d27332a16ac776/cgroup.procs
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d7575a9a863245702665a02fc3af5f784c19382aac8b880fe4d27332a16ac776/memory.stat
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4861952 rss
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d7575a9a863245702665a02fc3af5f784c19382aac8b880fe4d27332a16ac776/cpuacct.stat
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d7575a9a863245702665a02fc3af5f784c19382aac8b880fe4d27332a16ac776/cpuset.cpus
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d7575a9a863245702665a02fc3af5f784c19382aac8b880fe4d27332a16ac776/blkio.io_service_bytes
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Mounting task input collection to get interval_list
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Interval_List collection mounted at /keep/dcd0caa42cf81cfae2c2a7a7d6a5503a+94
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Total chunk length is 15618872
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Have 1 intervals
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:27:40 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 19 done, 3 running, 197 todo
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: cpu 0.5200 user 0.1500 sys 32 cpus -- interval 10.0001 seconds 0.4800 user 0.1400 sys
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr crunchstat: net:eth0 22051 tx 238193 rx
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Have 5400 potential reusable task outputs
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Have 5400 potentially reusable tasks
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Creating task to process interval: [11:57156751-72775622]
2016-12-07_01:27:44 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr No reusable JobTask matched key parameters [u'121_of_200', u'7770dae0a3517e6dca33c49e54e80074+14704', '11:57156751-72775622', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 stderr Ending task 1 successfully
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 notice: task is not in slurm queue but srun process 1517 has not exited
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr srun: Job step created
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 child 1854 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 success in 63 seconds
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297 24 task output (0 bytes): 
2016-12-07_01:27:45 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2vh6dr8bmvpcz1x-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2vh6dr8bmvpcz1x-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=21 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-2vh6dr8bmvpcz1x --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ce01352ba7531e731c62dd4893c8e96b52305c87d476f52d62cd7fc881566e07/cgroup.procs
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/ce01352ba7531e731c62dd4893c8e96b52305c87d476f52d62cd7fc881566e07/memory.stat
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5664768 rss
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ce01352ba7531e731c62dd4893c8e96b52305c87d476f52d62cd7fc881566e07/cpuacct.stat
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/ce01352ba7531e731c62dd4893c8e96b52305c87d476f52d62cd7fc881566e07/cpuset.cpus
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/ce01352ba7531e731c62dd4893c8e96b52305c87d476f52d62cd7fc881566e07/blkio.io_service_bytes
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Mounting task input collection to get interval_list
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Interval_List collection mounted at /keep/dbba1077cec452fe00018827674a6082+94
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Total chunk length is 15618872
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Have 1 intervals
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:27:52 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 20 done, 2 running, 198 todo
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Have 5400 potential reusable task outputs
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:27:55 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Have 5400 potentially reusable tasks
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Creating task to process interval: [11:25919007-41537878]
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr No reusable JobTask matched key parameters [u'119_of_200', u'e715710b28adf5479c2b4ee6a270292a+14700', '11:25919007-41537878', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr Ending task 1 successfully
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 52002816 rss
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: cpu 0.7800 user 0.1900 sys 32 cpus -- interval 10.0002 seconds 0.7400 user 0.1900 sys
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 stderr crunchstat: net:eth0 40222 tx 3250421 rx
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 child 1139 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 success in 97 seconds
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297 21 task output (0 bytes): 
2016-12-07_01:27:56 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr srun: Job step created
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-8zpggr6kb52w373-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-8zpggr6kb52w373-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=22 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-8zpggr6kb52w373 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/536010b0d337715a3127df53bf8bc73a590ac64777990472d1826405c4df4468/cgroup.procs
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/536010b0d337715a3127df53bf8bc73a590ac64777990472d1826405c4df4468/memory.stat
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5410816 rss
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/536010b0d337715a3127df53bf8bc73a590ac64777990472d1826405c4df4468/cpuacct.stat
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/536010b0d337715a3127df53bf8bc73a590ac64777990472d1826405c4df4468/cpuset.cpus
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/536010b0d337715a3127df53bf8bc73a590ac64777990472d1826405c4df4468/blkio.io_service_bytes
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Mounting task input collection to get interval_list
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Interval_List collection mounted at /keep/a19791935337560ca00961b9d0c50277+92
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Total chunk length is 15618872
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Have 1 intervals
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:28:03 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 21 done, 1 running, 199 todo
2016-12-07_01:28:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:28:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:28:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:28:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:28:06 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Have 5400 potential reusable task outputs
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 46022656 rss
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: cpu 0.6800 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.6400 user 0.1800 sys
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr crunchstat: net:eth0 35181 tx 3235991 rx
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Have 5400 potentially reusable tasks
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Creating task to process interval: [1:156188721-171807592]
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr No reusable JobTask matched key parameters [u'11_of_200', u'0407d47186859c17195bd2efb49797bb+14616', '1:156188721-171807592', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:28:07 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 stderr Ending task 1 successfully
2016-12-07_01:28:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 child 1517 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:28:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 success in 94 seconds
2016-12-07_01:28:08 z8ta6-8i9sb-r52unvxj6avgrio 58297 22 task output (0 bytes): 
2016-12-07_01:28:08 z8ta6-8i9sb-r52unvxj6avgrio 58297  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:28:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  Got 1 new tasks from API server.
2016-12-07_01:28:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  status: 22 done, 0 running, 200 todo
2016-12-07_01:28:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  collate
2016-12-07_01:28:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  collated output manifest text to send to API server is 0 bytes with access tokens
2016-12-07_01:28:14 z8ta6-8i9sb-r52unvxj6avgrio 58297  job output d41d8cd98f00b204e9800998ecf8427e+0
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  running from /var/www/arvados-api/shared/vendor_bundle/ruby/2.1.0/gems/arvados-cli-0.1.20161017193526/bin/crunch-job with arvados-cli Gem version(s) 0.1.20161017193526, 0.1.20160503204200, 0.1.20151207150126, 0.1.20151023190001
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  check slurm allocation
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  node humgen-05-15 - 8 slots
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  start
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  clean work dirs: start
2016-12-07_01:28:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','bash','-ec','-o','pipefail','mount -t fuse,fuse.keep | awk "(index(\\$3, \\"$CRUNCH_TMP\\") == 1){print \\$3}" | xargs -r -n 1 fusermount -u -z; sleep 1; rm -rf $JOB_WORK $CRUNCH_INSTALL $CRUNCH_TMP/task $CRUNCH_TMP/src* $CRUNCH_TMP/*.cid']
2016-12-07_01:28:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  clean work dirs: exit 0
2016-12-07_01:28:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  Install docker image 9289c4f1fce1d1dedf9cc7876c33f936+671
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  docker image hash is 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  load docker image: start
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr starting: ['srun','--nodelist=humgen-05-15','/bin/bash','-o','pipefail','-ec',' if /usr/bin/docker images -q --no-trunc --all | grep -xF 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 >/dev/null; then     exit 0 fi declare -a exit_codes=("${PIPESTATUS[@]}") if [ 0 != "${exit_codes[0]}" ]; then    exit "${exit_codes[0]}"  # `docker images` failed elif [ 1 != "${exit_codes[1]}" ]; then    exit "${exit_codes[1]}"  # `grep` encountered an error else    # Everything worked fine, but grep didn\'t find the image on this host.    arv-get 9289c4f1fce1d1dedf9cc7876c33f936\\+671\\/571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3\\.tar | /usr/bin/docker load fi ']
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  load docker image: exit 0
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  check --memory-swap feature: start
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr starting: ['srun','--nodes=1','/usr/bin/docker','run','--help']
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  check --memory-swap feature: exit 0
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  check whether default user is UID 0: start
2016-12-07_01:28:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr starting: ['srun','--nodes=1','/bin/sh','-ec','/usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21  571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 id --user']
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  check whether default user is UID 0: exit 0
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  Container will run with default user
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  Looking for version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 from repository jr17/hgi
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  Using local repository '/var/lib/arvados/internal.git'
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  Version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 is commit 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  run install script on all workers: start
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr starting: ['srun','--nodelist=humgen-05-15','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio','sh','-c','mkdir -p /data/crunch-tmp/crunch-job/opt && cd /data/crunch-tmp/crunch-job && perl -']
2016-12-07_01:28:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  stderr Extracting archive: 4b8cf8e1274f8d418cb494d3ea978152
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  run install script on all workers: exit 0
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  script gatk-combinegvcfs.py
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  script_version 794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  script_parameters {"interval_lists_collection":"4a72f79fd2ed51ac692bc326fb168f37+10043","reference_collection":"1b447d1aa006cde57df03042736750b9+7894","inputs_collection":"b0dd82bb014b6ab8e64968ebdc07aad9+2566847"}
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  runtime_constraints {"keep_cache_mb_per_task":19500,"max_tasks_per_node":8,"min_nodes":1,"docker_image":"mercury/gatk-3.6"}
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  start level 0 with 1 slots
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 0 done, 0 running, 1 todo
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 job_task z8ta6-ot0gb-jcys8e23gw0n907
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 child 5243 started on humgen-05-15.1
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.0.5243','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (1 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jcys8e23gw0n907-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-jcys8e23gw0n907-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jcys8e23gw0n907-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=0 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=0 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-jcys8e23gw0n907 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 0 done, 1 running, 0 todo
2016-12-07_01:28:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-jcys8e23gw0n907-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-jcys8e23gw0n907-0.cid --sig-proxy --memory=250853295k --memory-swap=284407723k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=0 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=0 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=1 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-jcys8e23gw0n907 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:28:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/f09f7b74a678a67c95ac4bb0f77808121353b067e4170c6fc01b657c7870b42e/cgroup.procs
2016-12-07_01:28:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/f09f7b74a678a67c95ac4bb0f77808121353b067e4170c6fc01b657c7870b42e/cpuacct.stat
2016-12-07_01:28:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 0.0000 user 0.0000 sys 0 cpus
2016-12-07_01:28:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/f09f7b74a678a67c95ac4bb0f77808121353b067e4170c6fc01b657c7870b42e/blkio.io_service_bytes
2016-12-07_01:28:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:28:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Processing last stream
2016-12-07_01:28:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Finalising stream .
2016-12-07_01:28:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 100_of_200
2016-12-07_01:28:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 100_of_200
2016-12-07_01:28:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 101_of_200
2016-12-07_01:28:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 101_of_200
2016-12-07_01:28:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 102_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 102_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 103_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 103_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 104_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 104_of_200
2016-12-07_01:28:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 105_of_200
2016-12-07_01:28:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 105_of_200
2016-12-07_01:28:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 106_of_200
2016-12-07_01:28:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 106_of_200
2016-12-07_01:28:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 107_of_200
2016-12-07_01:28:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 107_of_200
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 108_of_200
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 108_of_200
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 109_of_200
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 109_of_200
2016-12-07_01:28:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 10_of_200
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/f09f7b74a678a67c95ac4bb0f77808121353b067e4170c6fc01b657c7870b42e/memory.stat
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 160747520 rss
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/f09f7b74a678a67c95ac4bb0f77808121353b067e4170c6fc01b657c7870b42e/cpuset.cpus
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 4.6400 user 0.4200 sys 32 cpus -- interval 10.0004 seconds 4.6400 user 0.4200 sys
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 259427 tx 3671070 rx
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 10_of_200
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 110_of_200
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 110_of_200
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 111_of_200
2016-12-07_01:28:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 111_of_200
2016-12-07_01:28:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 112_of_200
2016-12-07_01:28:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 112_of_200
2016-12-07_01:28:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 113_of_200
2016-12-07_01:28:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 113_of_200
2016-12-07_01:28:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 114_of_200
2016-12-07_01:28:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 114_of_200
2016-12-07_01:28:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 115_of_200
2016-12-07_01:28:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 115_of_200
2016-12-07_01:28:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 116_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 116_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 117_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 117_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 118_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 118_of_200
2016-12-07_01:28:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 119_of_200
2016-12-07_01:28:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 119_of_200
2016-12-07_01:28:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 11_of_200
2016-12-07_01:28:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 11_of_200
2016-12-07_01:28:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 120_of_200
2016-12-07_01:28:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 120_of_200
2016-12-07_01:28:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 121_of_200
2016-12-07_01:28:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 121_of_200
2016-12-07_01:28:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 122_of_200
2016-12-07_01:28:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 122_of_200
2016-12-07_01:28:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 123_of_200
2016-12-07_01:28:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 123_of_200
2016-12-07_01:28:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 124_of_200
2016-12-07_01:28:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 124_of_200
2016-12-07_01:28:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 125_of_200
2016-12-07_01:28:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 125_of_200
2016-12-07_01:28:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 126_of_200
2016-12-07_01:28:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 126_of_200
2016-12-07_01:28:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 127_of_200
2016-12-07_01:28:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 127_of_200
2016-12-07_01:28:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 128_of_200
2016-12-07_01:28:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 128_of_200
2016-12-07_01:28:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 129_of_200
2016-12-07_01:28:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 129_of_200
2016-12-07_01:28:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 12_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 12_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 130_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 130_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 131_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 131_of_200
2016-12-07_01:28:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 132_of_200
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 177774592 rss
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 8.1800 user 0.6300 sys 32 cpus -- interval 10.0000 seconds 3.5400 user 0.2100 sys
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 856713 tx 4679854 rx -- interval 10.0000 seconds 597286 tx 1008784 rx
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 132_of_200
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 133_of_200
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 133_of_200
2016-12-07_01:28:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 134_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 134_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 135_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 135_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 136_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 136_of_200
2016-12-07_01:28:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 137_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 137_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 138_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 138_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 139_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 139_of_200
2016-12-07_01:28:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 13_of_200
2016-12-07_01:28:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 13_of_200
2016-12-07_01:28:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 140_of_200
2016-12-07_01:28:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 140_of_200
2016-12-07_01:28:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 141_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 141_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 142_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 142_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 143_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 143_of_200
2016-12-07_01:28:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 144_of_200
2016-12-07_01:28:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 144_of_200
2016-12-07_01:28:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 145_of_200
2016-12-07_01:28:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 145_of_200
2016-12-07_01:28:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 146_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 146_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 147_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 147_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 148_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 148_of_200
2016-12-07_01:28:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 149_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 149_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 14_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 14_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 150_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 150_of_200
2016-12-07_01:28:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 151_of_200
2016-12-07_01:28:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 151_of_200
2016-12-07_01:28:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 152_of_200
2016-12-07_01:28:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 152_of_200
2016-12-07_01:28:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 153_of_200
2016-12-07_01:28:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 153_of_200
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 154_of_200
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 154_of_200
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 155_of_200
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 155_of_200
2016-12-07_01:28:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 156_of_200
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 190222336 rss
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 11.6600 user 0.7800 sys 32 cpus -- interval 10.0000 seconds 3.4800 user 0.1500 sys
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 1473400 tx 5726521 rx -- interval 10.0000 seconds 616687 tx 1046667 rx
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 156_of_200
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 157_of_200
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 157_of_200
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 158_of_200
2016-12-07_01:28:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 158_of_200
2016-12-07_01:28:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 159_of_200
2016-12-07_01:28:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 159_of_200
2016-12-07_01:28:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 15_of_200
2016-12-07_01:28:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 15_of_200
2016-12-07_01:28:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 160_of_200
2016-12-07_01:28:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 160_of_200
2016-12-07_01:28:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 161_of_200
2016-12-07_01:28:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 161_of_200
2016-12-07_01:28:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 162_of_200
2016-12-07_01:28:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 162_of_200
2016-12-07_01:28:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 163_of_200
2016-12-07_01:28:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 163_of_200
2016-12-07_01:28:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 164_of_200
2016-12-07_01:28:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 164_of_200
2016-12-07_01:28:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 165_of_200
2016-12-07_01:28:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 165_of_200
2016-12-07_01:28:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 166_of_200
2016-12-07_01:28:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 166_of_200
2016-12-07_01:28:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 167_of_200
2016-12-07_01:28:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 167_of_200
2016-12-07_01:28:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 168_of_200
2016-12-07_01:29:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 168_of_200
2016-12-07_01:29:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 169_of_200
2016-12-07_01:29:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 169_of_200
2016-12-07_01:29:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 16_of_200
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 16_of_200
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 170_of_200
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 170_of_200
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 171_of_200
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 171_of_200
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 172_of_200
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 214007808 rss
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 13.9400 user 0.9300 sys 32 cpus -- interval 10.0000 seconds 2.2800 user 0.1500 sys
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 1899019 tx 6444281 rx -- interval 10.0000 seconds 425619 tx 717760 rx
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 172_of_200
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 173_of_200
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 173_of_200
2016-12-07_01:29:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 174_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 174_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 175_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 175_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 176_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 176_of_200
2016-12-07_01:29:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 177_of_200
2016-12-07_01:29:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 177_of_200
2016-12-07_01:29:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 178_of_200
2016-12-07_01:29:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 178_of_200
2016-12-07_01:29:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 179_of_200
2016-12-07_01:29:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 179_of_200
2016-12-07_01:29:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 17_of_200
2016-12-07_01:29:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 17_of_200
2016-12-07_01:29:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 180_of_200
2016-12-07_01:29:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 180_of_200
2016-12-07_01:29:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 181_of_200
2016-12-07_01:29:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 181_of_200
2016-12-07_01:29:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 182_of_200
2016-12-07_01:29:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 182_of_200
2016-12-07_01:29:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 183_of_200
2016-12-07_01:29:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 183_of_200
2016-12-07_01:29:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 184_of_200
2016-12-07_01:29:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 184_of_200
2016-12-07_01:29:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 185_of_200
2016-12-07_01:29:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 185_of_200
2016-12-07_01:29:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 186_of_200
2016-12-07_01:29:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 186_of_200
2016-12-07_01:29:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 187_of_200
2016-12-07_01:29:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 187_of_200
2016-12-07_01:29:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 188_of_200
2016-12-07_01:29:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 188_of_200
2016-12-07_01:29:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 189_of_200
2016-12-07_01:29:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 189_of_200
2016-12-07_01:29:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 18_of_200
2016-12-07_01:29:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 18_of_200
2016-12-07_01:29:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 190_of_200
2016-12-07_01:29:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 190_of_200
2016-12-07_01:29:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 191_of_200
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 191_of_200
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 192_of_200
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 192_of_200
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 193_of_200
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:12 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 193_of_200
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 194_of_200
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 241840128 rss
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 17.0800 user 1.1000 sys 32 cpus -- interval 10.0000 seconds 3.1400 user 0.1700 sys
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 2469397 tx 7411041 rx -- interval 10.0000 seconds 570378 tx 966760 rx
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 194_of_200
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 195_of_200
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 195_of_200
2016-12-07_01:29:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 196_of_200
2016-12-07_01:29:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 196_of_200
2016-12-07_01:29:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 197_of_200
2016-12-07_01:29:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 197_of_200
2016-12-07_01:29:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 198_of_200
2016-12-07_01:29:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 198_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 199_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 199_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 19_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 19_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 1_of_200
2016-12-07_01:29:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 1_of_200
2016-12-07_01:29:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 200_of_200
2016-12-07_01:29:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 200_of_200
2016-12-07_01:29:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 20_of_200
2016-12-07_01:29:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 20_of_200
2016-12-07_01:29:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 21_of_200
2016-12-07_01:29:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 21_of_200
2016-12-07_01:29:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 22_of_200
2016-12-07_01:29:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 22_of_200
2016-12-07_01:29:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 23_of_200
2016-12-07_01:29:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 23_of_200
2016-12-07_01:29:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 24_of_200
2016-12-07_01:29:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 24_of_200
2016-12-07_01:29:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 25_of_200
2016-12-07_01:29:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 25_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 26_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 26_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 27_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 27_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 28_of_200
2016-12-07_01:29:19 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 28_of_200
2016-12-07_01:29:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 29_of_200
2016-12-07_01:29:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 29_of_200
2016-12-07_01:29:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 2_of_200
2016-12-07_01:29:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 2_of_200
2016-12-07_01:29:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 30_of_200
2016-12-07_01:29:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 30_of_200
2016-12-07_01:29:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 31_of_200
2016-12-07_01:29:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 31_of_200
2016-12-07_01:29:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 32_of_200
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 32_of_200
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 33_of_200
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 33_of_200
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 34_of_200
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 34_of_200
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 270934016 rss
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 20.3600 user 1.2600 sys 32 cpus -- interval 10.0000 seconds 3.2800 user 0.1600 sys
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 3060506 tx 8413678 rx -- interval 10.0000 seconds 591109 tx 1002637 rx
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 35_of_200
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 35_of_200
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 36_of_200
2016-12-07_01:29:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 36_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 37_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 37_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 38_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 38_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 39_of_200
2016-12-07_01:29:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 39_of_200
2016-12-07_01:29:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 3_of_200
2016-12-07_01:29:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 3_of_200
2016-12-07_01:29:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 40_of_200
2016-12-07_01:29:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 40_of_200
2016-12-07_01:29:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 41_of_200
2016-12-07_01:29:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 41_of_200
2016-12-07_01:29:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 42_of_200
2016-12-07_01:29:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 42_of_200
2016-12-07_01:29:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 43_of_200
2016-12-07_01:29:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 43_of_200
2016-12-07_01:29:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 44_of_200
2016-12-07_01:29:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 44_of_200
2016-12-07_01:29:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 45_of_200
2016-12-07_01:29:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 45_of_200
2016-12-07_01:29:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 46_of_200
2016-12-07_01:29:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 46_of_200
2016-12-07_01:29:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 47_of_200
2016-12-07_01:29:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 47_of_200
2016-12-07_01:29:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 48_of_200
2016-12-07_01:29:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 48_of_200
2016-12-07_01:29:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 49_of_200
2016-12-07_01:29:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 49_of_200
2016-12-07_01:29:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 4_of_200
2016-12-07_01:29:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 4_of_200
2016-12-07_01:29:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 50_of_200
2016-12-07_01:29:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 50_of_200
2016-12-07_01:29:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 51_of_200
2016-12-07_01:29:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 51_of_200
2016-12-07_01:29:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 52_of_200
2016-12-07_01:29:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 52_of_200
2016-12-07_01:29:31 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 53_of_200
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 53_of_200
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 54_of_200
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 54_of_200
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 55_of_200
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 298774528 rss
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 23.3200 user 1.4200 sys 32 cpus -- interval 10.0000 seconds 2.9600 user 0.1600 sys
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 3583227 tx 9307377 rx -- interval 10.0000 seconds 522721 tx 893699 rx
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 55_of_200
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 56_of_200
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 56_of_200
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 57_of_200
2016-12-07_01:29:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 57_of_200
2016-12-07_01:29:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 58_of_200
2016-12-07_01:29:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 58_of_200
2016-12-07_01:29:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 59_of_200
2016-12-07_01:29:34 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 59_of_200
2016-12-07_01:29:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 5_of_200
2016-12-07_01:29:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 5_of_200
2016-12-07_01:29:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 60_of_200
2016-12-07_01:29:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 60_of_200
2016-12-07_01:29:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 61_of_200
2016-12-07_01:29:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 61_of_200
2016-12-07_01:29:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 62_of_200
2016-12-07_01:29:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 62_of_200
2016-12-07_01:29:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 63_of_200
2016-12-07_01:29:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 63_of_200
2016-12-07_01:29:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 64_of_200
2016-12-07_01:29:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 64_of_200
2016-12-07_01:29:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 65_of_200
2016-12-07_01:29:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 65_of_200
2016-12-07_01:29:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 66_of_200
2016-12-07_01:29:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 66_of_200
2016-12-07_01:29:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 67_of_200
2016-12-07_01:29:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 67_of_200
2016-12-07_01:29:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 68_of_200
2016-12-07_01:29:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 68_of_200
2016-12-07_01:29:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 69_of_200
2016-12-07_01:29:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 69_of_200
2016-12-07_01:29:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 6_of_200
2016-12-07_01:29:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 6_of_200
2016-12-07_01:29:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 70_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 70_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 71_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 71_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 72_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 72_of_200
2016-12-07_01:29:41 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 73_of_200
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 73_of_200
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 74_of_200
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 74_of_200
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 75_of_200
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 309661696 rss
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 26.8400 user 1.7000 sys 32 cpus -- interval 9.9999 seconds 3.5200 user 0.2800 sys
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 4141127 tx 10205868 rx -- interval 9.9999 seconds 557900 tx 898491 rx
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 75_of_200
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 76_of_200
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 76_of_200
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 77_of_200
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 77_of_200
2016-12-07_01:29:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 78_of_200
2016-12-07_01:29:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 78_of_200
2016-12-07_01:29:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 79_of_200
2016-12-07_01:29:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 79_of_200
2016-12-07_01:29:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 7_of_200
2016-12-07_01:29:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 7_of_200
2016-12-07_01:29:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 80_of_200
2016-12-07_01:29:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 80_of_200
2016-12-07_01:29:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 81_of_200
2016-12-07_01:29:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 81_of_200
2016-12-07_01:29:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 82_of_200
2016-12-07_01:29:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 82_of_200
2016-12-07_01:29:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 83_of_200
2016-12-07_01:29:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 83_of_200
2016-12-07_01:29:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 84_of_200
2016-12-07_01:29:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 84_of_200
2016-12-07_01:29:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 85_of_200
2016-12-07_01:29:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 85_of_200
2016-12-07_01:29:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 86_of_200
2016-12-07_01:29:47 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 86_of_200
2016-12-07_01:29:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 87_of_200
2016-12-07_01:29:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 87_of_200
2016-12-07_01:29:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 88_of_200
2016-12-07_01:29:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 88_of_200
2016-12-07_01:29:48 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 89_of_200
2016-12-07_01:29:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 89_of_200
2016-12-07_01:29:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 8_of_200
2016-12-07_01:29:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 8_of_200
2016-12-07_01:29:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 90_of_200
2016-12-07_01:29:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 90_of_200
2016-12-07_01:29:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 91_of_200
2016-12-07_01:29:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 91_of_200
2016-12-07_01:29:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 92_of_200
2016-12-07_01:29:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 92_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 93_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 93_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 94_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 94_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 95_of_200
2016-12-07_01:29:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 95_of_200
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 96_of_200
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 96_of_200
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 97_of_200
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 97_of_200
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcalls 0 put 0 get -- interval 10.0000 seconds 0 put 0 get
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:keep0 0 tx 0 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: keepcache 0 hit 0 miss -- interval 10.0000 seconds 0 hit 0 miss
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: fuseops 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: blkio:0:0 0 write 0 read -- interval 10.0000 seconds 0 write 0 read
2016-12-07_01:29:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 98_of_200
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: mem 950272 cache 0 swap 0 pgmajfault 309698560 rss
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: cpu 30.0400 user 1.8600 sys 32 cpus -- interval 10.0000 seconds 3.2000 user 0.1600 sys
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr crunchstat: net:eth0 4712642 tx 11203923 rx -- interval 10.0000 seconds 571515 tx 998055 rx
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 98_of_200
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 99_of_200
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 99_of_200
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Have 42 gVCFs in group 9_of_200
2016-12-07_01:29:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Creating task to process 9_of_200
2016-12-07_01:29:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 stderr Ending task 0 successfully
2016-12-07_01:29:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 child 5243 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:29:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 success in 92 seconds
2016-12-07_01:29:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 0 task output (0 bytes): 
2016-12-07_01:29:55 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:30:04 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 200 new tasks from API server.
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 1 done, 0 running, 200 todo
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  wait for last 0 children to finish
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  start level 1 with 8 slots
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 1 done, 0 running, 200 todo
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 job_task z8ta6-ot0gb-hizpczdtbjel8pt
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 child 8311 started on humgen-05-15.1
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 job_task z8ta6-ot0gb-t8qg32goo9pc50a
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 child 8322 started on humgen-05-15.2
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 job_task z8ta6-ot0gb-gaupyyeay9of4sb
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 child 8327 started on humgen-05-15.3
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 job_task z8ta6-ot0gb-9d2ze0wl27bbr45
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 child 8335 started on humgen-05-15.4
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 job_task z8ta6-ot0gb-iyvt2bz6ed4h7sk
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 child 8339 started on humgen-05-15.5
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 job_task z8ta6-ot0gb-5h8cpstnmzy5a9s
2016-12-07_01:30:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 child 8353 started on humgen-05-15.6
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 job_task z8ta6-ot0gb-qw3ghu26grvq3m3
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 child 8357 started on humgen-05-15.7
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 job_task z8ta6-ot0gb-2q378ahzc2xpimi
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 child 8361 started on humgen-05-15.8
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.1.8311','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-hizpczdtbjel8pt-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-hizpczdtbjel8pt-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-hizpczdtbjel8pt-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=1 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-hizpczdtbjel8pt --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-hizpczdtbjel8pt-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-hizpczdtbjel8pt-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=1 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-hizpczdtbjel8pt --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/0409462a17067068bef96d8d33ed30219d6ad1d93a8c1b71b81d97ae16a99570/cgroup.procs
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/0409462a17067068bef96d8d33ed30219d6ad1d93a8c1b71b81d97ae16a99570/memory.stat
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 2867200 rss
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/0409462a17067068bef96d8d33ed30219d6ad1d93a8c1b71b81d97ae16a99570/cpuacct.stat
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/0409462a17067068bef96d8d33ed30219d6ad1d93a8c1b71b81d97ae16a99570/cpuset.cpus
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/0409462a17067068bef96d8d33ed30219d6ad1d93a8c1b71b81d97ae16a99570/blkio.io_service_bytes
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.2.8322','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t8qg32goo9pc50a-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-t8qg32goo9pc50a-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t8qg32goo9pc50a-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=2 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-t8qg32goo9pc50a --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.3.8327','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-gaupyyeay9of4sb-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-gaupyyeay9of4sb-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-gaupyyeay9of4sb-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=3 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-gaupyyeay9of4sb --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.4.8335','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9d2ze0wl27bbr45-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-9d2ze0wl27bbr45-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9d2ze0wl27bbr45-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=4 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-9d2ze0wl27bbr45 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.5.8339','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-iyvt2bz6ed4h7sk-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-iyvt2bz6ed4h7sk-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-iyvt2bz6ed4h7sk-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=5 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-iyvt2bz6ed4h7sk --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.6.8353','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5h8cpstnmzy5a9s-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-5h8cpstnmzy5a9s-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5h8cpstnmzy5a9s-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=6 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-5h8cpstnmzy5a9s --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.7.8357','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qw3ghu26grvq3m3-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-qw3ghu26grvq3m3-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qw3ghu26grvq3m3-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=7 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-qw3ghu26grvq3m3 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.8.8361','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2q378ahzc2xpimi-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-2q378ahzc2xpimi-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-2q378ahzc2xpimi-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=8 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-2q378ahzc2xpimi --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:06 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 1 done, 8 running, 192 todo
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Mounting task input collection to get interval_list
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Interval_List collection mounted at /keep/812316f6c95d8641412912523842bbfc+94
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Total chunk length is 15618872
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Have 1 intervals
2016-12-07_01:30:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:30:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:30:09 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:30:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: blkio:0:0 0 write 12523 read -- interval 10.0000 seconds 0 write 12523 read
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Have 5400 potential reusable task outputs
2016-12-07_01:30:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Have 5400 potentially reusable tasks
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Creating task to process interval: [9:6148617-21767488]
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr No reusable JobTask matched key parameters [u'100_of_200', u'a241297995deeab501f9bd99dc533315+14700', '9:6148617-21767488', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 48857088 rss
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: cpu 0.7000 user 0.2100 sys 32 cpus -- interval 10.0001 seconds 0.6800 user 0.2100 sys
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr crunchstat: net:eth0 35651 tx 3238200 rx
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 stderr Ending task 1 successfully
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 child 8311 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 success in 11 seconds
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970 1 task output (0 bytes): 
2016-12-07_01:30:16 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 2 done, 7 running, 193 todo
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 job_task z8ta6-ot0gb-0rkh929u4mx4kvh
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 child 9178 started on humgen-05-15.1
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.9.9178','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0rkh929u4mx4kvh-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0rkh929u4mx4kvh-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0rkh929u4mx4kvh-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=9 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-0rkh929u4mx4kvh --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-t8qg32goo9pc50a-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t8qg32goo9pc50a-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.2.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=2 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.2 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=2 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.2 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-t8qg32goo9pc50a --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8621d8d616bb073a6d72f06b4ada591eae3120d3a4fd3a506bab81a0a6919b5c/cgroup.procs
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8621d8d616bb073a6d72f06b4ada591eae3120d3a4fd3a506bab81a0a6919b5c/memory.stat
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 3014656 rss
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8621d8d616bb073a6d72f06b4ada591eae3120d3a4fd3a506bab81a0a6919b5c/cpuacct.stat
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8621d8d616bb073a6d72f06b4ada591eae3120d3a4fd3a506bab81a0a6919b5c/cpuset.cpus
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:30:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8621d8d616bb073a6d72f06b4ada591eae3120d3a4fd3a506bab81a0a6919b5c/blkio.io_service_bytes
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Mounting task input collection to get interval_list
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Interval_List collection mounted at /keep/6af141a235ccb024f6a32fc12ee98dcf+94
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Total chunk length is 15618872
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Have 1 intervals
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:24 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 2 done, 8 running, 192 todo
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24092672 rss
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: cpu 0.4900 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.4700 user 0.1800 sys
2016-12-07_01:30:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr crunchstat: net:eth0 21807 tx 238280 rx
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Have 5400 potential reusable task outputs
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Have 5400 potentially reusable tasks
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Creating task to process interval: [9:21767489-37386360]
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr No reusable JobTask matched key parameters [u'101_of_200', u'1b848222e216cbae87e6932d5820a285+14700', '9:21767489-37386360', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 stderr Ending task 1 successfully
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 child 8322 on humgen-05-15.2 exit 0 success=true
2016-12-07_01:30:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 success in 23 seconds
2016-12-07_01:30:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 2 task output (0 bytes): 
2016-12-07_01:30:29 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 3 done, 7 running, 193 todo
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 job_task z8ta6-ot0gb-04fu0g2l0893trg
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 child 9506 started on humgen-05-15.2
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.10.9506','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-04fu0g2l0893trg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-04fu0g2l0893trg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-04fu0g2l0893trg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=10 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-04fu0g2l0893trg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr srun: Job step created
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-gaupyyeay9of4sb-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-gaupyyeay9of4sb-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=3 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-gaupyyeay9of4sb --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d78569bda201583317540e52c3114637b00b0246e0874874cd3f0d0c0888e076/cgroup.procs
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d78569bda201583317540e52c3114637b00b0246e0874874cd3f0d0c0888e076/cpuacct.stat
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr crunchstat: cpu 0.0100 user 0.0000 sys 0 cpus
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d78569bda201583317540e52c3114637b00b0246e0874874cd3f0d0c0888e076/blkio.io_service_bytes
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Mounting task input collection to get interval_list
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Interval_List collection mounted at /keep/2e7cf8719961c4cc5b2ffdd92ff99dd0+94
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Total chunk length is 15618872
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Have 1 intervals
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:30:35 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:30:36 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 3 done, 8 running, 192 todo
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Have 5400 potential reusable task outputs
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Have 5400 potentially reusable tasks
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Creating task to process interval: [9:37386361-53005232]
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr No reusable JobTask matched key parameters [u'102_of_200', u'0b01f4aa638023c7844e14aa840fe4cc+14532', '9:37386361-53005232', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:30:38 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 stderr Ending task 1 successfully
2016-12-07_01:30:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 child 8327 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:30:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 success in 34 seconds
2016-12-07_01:30:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 3 task output (0 bytes): 
2016-12-07_01:30:39 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:30:45 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 4 done, 7 running, 193 todo
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 job_task z8ta6-ot0gb-tgduaa5l7ohexgw
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 child 9860 started on humgen-05-15.3
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.11.9860','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-tgduaa5l7ohexgw-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-tgduaa5l7ohexgw-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-tgduaa5l7ohexgw-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=11 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-tgduaa5l7ohexgw --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr srun: Job step created
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-iyvt2bz6ed4h7sk-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-iyvt2bz6ed4h7sk-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=5 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-iyvt2bz6ed4h7sk --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d5e93ba2cea615515fbebb655346323cc5e4cec5afedb75c13ad5ab461442b6d/cgroup.procs
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d5e93ba2cea615515fbebb655346323cc5e4cec5afedb75c13ad5ab461442b6d/memory.stat
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4927488 rss
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d5e93ba2cea615515fbebb655346323cc5e4cec5afedb75c13ad5ab461442b6d/cpuacct.stat
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d5e93ba2cea615515fbebb655346323cc5e4cec5afedb75c13ad5ab461442b6d/cpuset.cpus
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d5e93ba2cea615515fbebb655346323cc5e4cec5afedb75c13ad5ab461442b6d/blkio.io_service_bytes
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Mounting task input collection to get interval_list
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Interval_List collection mounted at /keep/61144aa3079bc748eb474251bb12d2af+94
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Total chunk length is 15618872
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Have 1 intervals
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:30:46 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 4 done, 8 running, 192 todo
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: blkio:0:0 0 write 12524 read -- interval 10.0000 seconds 0 write 12524 read
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Have 5400 potential reusable task outputs
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 46034944 rss
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: cpu 0.6800 user 0.2000 sys 32 cpus -- interval 10.0002 seconds 0.6600 user 0.2000 sys
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr crunchstat: net:eth0 34323 tx 3236409 rx
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Have 5400 potentially reusable tasks
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Creating task to process interval: [9:68624105-84242976]
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr No reusable JobTask matched key parameters [u'104_of_200', u'0e7a8a6d12ed4d9a7f7a47c972f98c08+14700', '9:68624105-84242976', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:30:49 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 stderr Ending task 1 successfully
2016-12-07_01:30:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 child 8339 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:30:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 success in 45 seconds
2016-12-07_01:30:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 5 task output (0 bytes): 
2016-12-07_01:30:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 5 done, 7 running, 193 todo
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 job_task z8ta6-ot0gb-t2losx80lggglyb
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 child 10215 started on humgen-05-15.5
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr srun: Job step created
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0rkh929u4mx4kvh-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0rkh929u4mx4kvh-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=9 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-0rkh929u4mx4kvh --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/607b276ad9f1f43ae345baeda8de73554888b0bc9f55394c20440f7d7be7d730/cgroup.procs
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/607b276ad9f1f43ae345baeda8de73554888b0bc9f55394c20440f7d7be7d730/memory.stat
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4640768 rss
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/607b276ad9f1f43ae345baeda8de73554888b0bc9f55394c20440f7d7be7d730/cpuacct.stat
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/607b276ad9f1f43ae345baeda8de73554888b0bc9f55394c20440f7d7be7d730/cpuset.cpus
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/607b276ad9f1f43ae345baeda8de73554888b0bc9f55394c20440f7d7be7d730/blkio.io_service_bytes
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Mounting task input collection to get interval_list
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Interval_List collection mounted at /keep/64696f9b403b7674c728386882cbd3b7+94
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Total chunk length is 15498872
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Have 1 intervals
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.12.10215','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t2losx80lggglyb-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-t2losx80lggglyb-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t2losx80lggglyb-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=12 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-t2losx80lggglyb --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:30:57 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 5 done, 8 running, 192 todo
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: blkio:0:0 0 write 12564 read -- interval 10.0000 seconds 0 write 12564 read
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Have 5400 potential reusable task outputs
2016-12-07_01:31:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Have 5400 potentially reusable tasks
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Creating task to process interval: [9:131099593-141213431 10:1-5385033]
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr No reusable JobTask matched key parameters [u'108_of_200', u'7e64dee294f55ab4bfb56c1a154618f8+14704', '9:131099593-141213431 10:1-5385033', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 49082368 rss
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: cpu 0.7600 user 0.1800 sys 32 cpus -- interval 10.0002 seconds 0.7200 user 0.1700 sys
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr crunchstat: net:eth0 36730 tx 3244582 rx
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 stderr Ending task 1 successfully
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr srun: Job step created
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 child 9178 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 success in 38 seconds
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 9 task output (0 bytes): 
2016-12-07_01:31:01 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 6 done, 7 running, 193 todo
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 job_task z8ta6-ot0gb-6u2qax3rliz18jb
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 child 10526 started on humgen-05-15.1
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.13.10526','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-6u2qax3rliz18jb-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-6u2qax3rliz18jb-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-6u2qax3rliz18jb-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=13 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-6u2qax3rliz18jb --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-5h8cpstnmzy5a9s-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-5h8cpstnmzy5a9s-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=6 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-5h8cpstnmzy5a9s --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/81bac0be8cb151c27361fe76142b5153f35732f263a71633b21875a48d3e44d4/cgroup.procs
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/81bac0be8cb151c27361fe76142b5153f35732f263a71633b21875a48d3e44d4/memory.stat
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5849088 rss
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/81bac0be8cb151c27361fe76142b5153f35732f263a71633b21875a48d3e44d4/cpuacct.stat
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/81bac0be8cb151c27361fe76142b5153f35732f263a71633b21875a48d3e44d4/cpuset.cpus
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: cpu 0.0200 user 0.0200 sys 32 cpus
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/81bac0be8cb151c27361fe76142b5153f35732f263a71633b21875a48d3e44d4/blkio.io_service_bytes
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Mounting task input collection to get interval_list
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Interval_List collection mounted at /keep/4325c7dac5c39f2db578e849695a12aa+94
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Total chunk length is 15618872
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Have 1 intervals
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:31:08 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 6 done, 8 running, 192 todo
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Have 5400 potential reusable task outputs
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Have 5400 potentially reusable tasks
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Creating task to process interval: [9:84242977-99861848]
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr No reusable JobTask matched key parameters [u'105_of_200', u'3a3bb24e84ba30a06b3d46caa3b1c040+14700', '9:84242977-99861848', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 stderr Ending task 1 successfully
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr srun: Job step created
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 child 8353 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:31:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 success in 65 seconds
2016-12-07_01:31:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 6 task output (0 bytes): 
2016-12-07_01:31:11 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:31:17 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:31:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 notice: task is not in slurm queue but srun process 8361 has not exited
2016-12-07_01:31:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 notice: task is not in slurm queue but srun process 8335 has not exited
2016-12-07_01:31:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 notice: task is not in slurm queue but srun process 8357 has not exited
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 7 done, 7 running, 193 todo
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 job_task z8ta6-ot0gb-65d5grcm0p15z7j
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 child 10870 started on humgen-05-15.6
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-t2losx80lggglyb-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-t2losx80lggglyb-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=12 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-t2losx80lggglyb --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b/cgroup.procs
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b/memory.stat
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 8159232 rss
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b/cpuacct.stat
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b/cpuset.cpus
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: cpu 0.0500 user 0.0000 sys 32 cpus
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b/blkio.io_service_bytes
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Mounting task input collection to get interval_list
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Interval_List collection mounted at /keep/11837032d32bcfff7055a484f9c0ddb8+94
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Total chunk length is 15618872
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Have 1 intervals
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.14.10870','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-65d5grcm0p15z7j-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-65d5grcm0p15z7j-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-65d5grcm0p15z7j-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=14 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-65d5grcm0p15z7j --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:31:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 7 done, 8 running, 192 todo
2016-12-07_01:31:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Have 5400 potential reusable task outputs
2016-12-07_01:31:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Have 5400 potentially reusable tasks
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Creating task to process interval: [10:21003906-36622777]
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr No reusable JobTask matched key parameters [u'110_of_200', u'e20f5f00b25cd7e53f799770dbe9e308+14700', '10:21003906-36622777', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr Ending task 1 successfully
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: stats not available: stat memory.stat, statgroup memory, cid 40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b, parent docker, root /sys/fs/cgroup
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: stats not available: stat cpuacct.stat, statgroup cpuacct, cid 40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b, parent docker, root /sys/fs/cgroup
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: stats not available: stat blkio.io_service_bytes, statgroup blkio, cid 40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b, parent docker, root /sys/fs/cgroup
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 stderr crunchstat: notice: stats not available: stat cgroup.procs, statgroup cpuacct, cid 40f430805003f21547d09efd726df84a91f56fda332ccd25174379435c4b305b, parent docker, root /sys/fs/cgroup
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 child 10215 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 success in 24 seconds
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970 12 task output (0 bytes): 
2016-12-07_01:31:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 8 done, 7 running, 193 todo
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 job_task z8ta6-ot0gb-w4g13gcmycpxkrd
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 child 11199 started on humgen-05-15.5
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.15.11199','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w4g13gcmycpxkrd-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-w4g13gcmycpxkrd-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w4g13gcmycpxkrd-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=15 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-w4g13gcmycpxkrd --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr srun: Job step created
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-qw3ghu26grvq3m3-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qw3ghu26grvq3m3-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=7 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-qw3ghu26grvq3m3 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ae3df328ab3c9a464b741031f5bfca6ff0bc593ad117af0800c0eb4830295077/cgroup.procs
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/ae3df328ab3c9a464b741031f5bfca6ff0bc593ad117af0800c0eb4830295077/memory.stat
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4849664 rss
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/ae3df328ab3c9a464b741031f5bfca6ff0bc593ad117af0800c0eb4830295077/cpuacct.stat
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/ae3df328ab3c9a464b741031f5bfca6ff0bc593ad117af0800c0eb4830295077/cpuset.cpus
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/ae3df328ab3c9a464b741031f5bfca6ff0bc593ad117af0800c0eb4830295077/blkio.io_service_bytes
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Mounting task input collection to get interval_list
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Interval_List collection mounted at /keep/34f0119dab554faf0acfc04f3cae7c1d+94
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Total chunk length is 15618872
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:31:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Have 1 intervals
2016-12-07_01:31:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:31:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:31:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:31:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:31:29 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 8 done, 8 running, 192 todo
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: cpu 0.5100 user 0.1500 sys 32 cpus -- interval 10.0001 seconds 0.4700 user 0.1500 sys
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr crunchstat: net:eth0 21877 tx 238215 rx
2016-12-07_01:31:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Have 5400 potential reusable task outputs
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Have 5400 potentially reusable tasks
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Creating task to process interval: [9:99861849-115480720]
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr No reusable JobTask matched key parameters [u'106_of_200', u'dc8fa207ac10a02e00e1ba8ab1687ad9+14700', '9:99861849-115480720', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 stderr Ending task 1 successfully
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 child 8357 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 success in 87 seconds
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 7 task output (0 bytes): 
2016-12-07_01:31:33 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 9 done, 7 running, 193 todo
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 job_task z8ta6-ot0gb-mrupikb9g03ehny
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 child 11540 started on humgen-05-15.7
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr srun: Job step created
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-6u2qax3rliz18jb-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-6u2qax3rliz18jb-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=13 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-6u2qax3rliz18jb --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/c73bae116d24a06a1b3fb456b68de5186a6ba2dd90ba673d1078510b398d4904/cgroup.procs
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/c73bae116d24a06a1b3fb456b68de5186a6ba2dd90ba673d1078510b398d4904/memory.stat
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5693440 rss
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/c73bae116d24a06a1b3fb456b68de5186a6ba2dd90ba673d1078510b398d4904/cpuacct.stat
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/c73bae116d24a06a1b3fb456b68de5186a6ba2dd90ba673d1078510b398d4904/cpuset.cpus
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: cpu 0.0500 user 0.0000 sys 32 cpus
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/c73bae116d24a06a1b3fb456b68de5186a6ba2dd90ba673d1078510b398d4904/blkio.io_service_bytes
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Mounting task input collection to get interval_list
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Interval_List collection mounted at /keep/d6cf1afdecfe2ae2261292a571a3759b+94
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Total chunk length is 15618872
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Have 1 intervals
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.16.11540','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-mrupikb9g03ehny-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-mrupikb9g03ehny-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-mrupikb9g03ehny-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=16 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-mrupikb9g03ehny --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:31:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 9 done, 8 running, 192 todo
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Have 5400 potential reusable task outputs
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Have 5400 potentially reusable tasks
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Creating task to process interval: [10:36622778-52241649]
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr No reusable JobTask matched key parameters [u'111_of_200', u'94eedabc31fa41e0ae0f2bed3edca749+14699', '10:36622778-52241649', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 stderr Ending task 1 successfully
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 child 10526 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:31:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 success in 35 seconds
2016-12-07_01:31:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 13 task output (0 bytes): 
2016-12-07_01:31:44 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 killing orphaned srun process 8361 (task not in slurm queue, no stderr received in last 86s)
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 sending 2x signal 2 to pid 8361
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 killing orphaned srun process 8335 (task not in slurm queue, no stderr received in last 86s)
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 sending 2x signal 2 to pid 8335
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 notice: task is not in slurm queue but srun process 9506 has not exited
2016-12-07_01:31:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 10 done, 7 running, 193 todo
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 job_task z8ta6-ot0gb-xn87jpqo3q9dql1
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 child 12176 started on humgen-05-15.1
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.17.12176','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xn87jpqo3q9dql1-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xn87jpqo3q9dql1-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xn87jpqo3q9dql1-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=17 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-xn87jpqo3q9dql1 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr srun: Job step created
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-tgduaa5l7ohexgw-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-tgduaa5l7ohexgw-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=11 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-tgduaa5l7ohexgw --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/55badd750a545955ee5c11833ab6aa6a295ef05e1478ae677d9b19cf0fc60925/cgroup.procs
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/55badd750a545955ee5c11833ab6aa6a295ef05e1478ae677d9b19cf0fc60925/memory.stat
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4898816 rss
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/55badd750a545955ee5c11833ab6aa6a295ef05e1478ae677d9b19cf0fc60925/cpuacct.stat
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/55badd750a545955ee5c11833ab6aa6a295ef05e1478ae677d9b19cf0fc60925/cpuset.cpus
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/55badd750a545955ee5c11833ab6aa6a295ef05e1478ae677d9b19cf0fc60925/blkio.io_service_bytes
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Mounting task input collection to get interval_list
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Interval_List collection mounted at /keep/47924fc036d6a931b5f0cf53440e0969+92
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Total chunk length is 15618872
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Have 1 intervals
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 child 8335 on humgen-05-15.4 exit 1 success=
2016-12-07_01:31:51 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:31:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 failure (#1, temporary) after 106 seconds
2016-12-07_01:31:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 4 task output (0 bytes): 
2016-12-07_01:31:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 child 8361 on humgen-05-15.8 exit 1 success=
2016-12-07_01:31:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:31:54 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 failure (#1, temporary) after 107 seconds
2016-12-07_01:31:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 8 task output (0 bytes): 
2016-12-07_01:31:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 child 9860 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:31:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 success in 69 seconds
2016-12-07_01:31:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 11 task output (0 bytes): 
2016-12-07_01:31:55 z8ta6-8i9sb-r52unvxj6avgrio 4970  3 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 11 done, 5 running, 195 todo
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 job_task z8ta6-ot0gb-ljma8zh3j9drwfa
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 child 12495 started on humgen-05-15.4
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 job_task z8ta6-ot0gb-zex553q8rtr26xg
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 child 12499 started on humgen-05-15.8
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 job_task z8ta6-ot0gb-9pmmsjzn1rgx4qz
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 child 12503 started on humgen-05-15.3
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.18.12495','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ljma8zh3j9drwfa-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ljma8zh3j9drwfa-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ljma8zh3j9drwfa-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=18 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ljma8zh3j9drwfa --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.19.12499','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zex553q8rtr26xg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-zex553q8rtr26xg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zex553q8rtr26xg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=19 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-zex553q8rtr26xg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr srun: Job step created
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-w4g13gcmycpxkrd-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-w4g13gcmycpxkrd-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=15 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-w4g13gcmycpxkrd --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/48f48b69c44f4c65605fdc376c0d92041b2e2d13b77273168c3f9835d979d76c/cgroup.procs
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/48f48b69c44f4c65605fdc376c0d92041b2e2d13b77273168c3f9835d979d76c/memory.stat
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 1626112 rss
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/48f48b69c44f4c65605fdc376c0d92041b2e2d13b77273168c3f9835d979d76c/cpuacct.stat
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/48f48b69c44f4c65605fdc376c0d92041b2e2d13b77273168c3f9835d979d76c/cpuset.cpus
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: cpu 0.0200 user 0.0000 sys 32 cpus
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/48f48b69c44f4c65605fdc376c0d92041b2e2d13b77273168c3f9835d979d76c/blkio.io_service_bytes
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Mounting task input collection to get interval_list
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Interval_List collection mounted at /keep/f73eab8bfca061b956c4cbaa59d62ccd+94
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Total chunk length is 15618872
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Have 1 intervals
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.20.12503','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9pmmsjzn1rgx4qz-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-9pmmsjzn1rgx4qz-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9pmmsjzn1rgx4qz-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=20 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-9pmmsjzn1rgx4qz --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:02 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 11 done, 8 running, 192 todo
2016-12-07_01:32:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:32:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:32:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:32:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:32:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:32:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:32:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: cpu 0.5200 user 0.1500 sys 32 cpus -- interval 10.0001 seconds 0.5000 user 0.1500 sys
2016-12-07_01:32:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr crunchstat: net:eth0 21877 tx 238084 rx
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Have 5400 potential reusable task outputs
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Have 5400 potentially reusable tasks
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Creating task to process interval: [10:67860522-83479393]
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr No reusable JobTask matched key parameters [u'113_of_200', u'45d9d4c732c4ecae90a7f5ec11993512+14700', '10:67860522-83479393', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 stderr Ending task 1 successfully
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 child 11199 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 success in 39 seconds
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 15 task output (0 bytes): 
2016-12-07_01:32:07 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 12 done, 7 running, 193 todo
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 job_task z8ta6-ot0gb-zi4wk7zwhebjuoo
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 child 12866 started on humgen-05-15.5
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.21.12866','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zi4wk7zwhebjuoo-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-zi4wk7zwhebjuoo-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zi4wk7zwhebjuoo-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=21 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-zi4wk7zwhebjuoo --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr srun: Job step created
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-65d5grcm0p15z7j-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-65d5grcm0p15z7j-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=14 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-65d5grcm0p15z7j --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/809c6d8a94d46b3aa02cff9a42d2d506d83a96bac03e507e8e67de24eb892637/cgroup.procs
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/809c6d8a94d46b3aa02cff9a42d2d506d83a96bac03e507e8e67de24eb892637/memory.stat
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 7585792 rss
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/809c6d8a94d46b3aa02cff9a42d2d506d83a96bac03e507e8e67de24eb892637/cpuacct.stat
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/809c6d8a94d46b3aa02cff9a42d2d506d83a96bac03e507e8e67de24eb892637/cpuset.cpus
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/809c6d8a94d46b3aa02cff9a42d2d506d83a96bac03e507e8e67de24eb892637/blkio.io_service_bytes
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Mounting task input collection to get interval_list
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Interval_List collection mounted at /keep/fe3ded28e7bf89d25f8d4e08ecc0d6c5+94
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Total chunk length is 15618872
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Have 1 intervals
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:32:14 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 12 done, 8 running, 192 todo
2016-12-07_01:32:17 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Have 5400 potential reusable task outputs
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Have 5400 potentially reusable tasks
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Creating task to process interval: [10:52241650-67860521]
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr No reusable JobTask matched key parameters [u'112_of_200', u'9e1996fed17b6a2fca279a862266ce9d+14700', '10:52241650-67860521', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr Ending task 1 successfully
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 47923200 rss
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: cpu 0.7700 user 0.2100 sys 32 cpus -- interval 10.0002 seconds 0.7400 user 0.2100 sys
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 stderr crunchstat: net:eth0 39639 tx 3252569 rx
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 child 10870 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 success in 60 seconds
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970 14 task output (0 bytes): 
2016-12-07_01:32:18 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 killing orphaned srun process 9506 (task not in slurm queue, no stderr received in last 99s)
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 sending 2x signal 2 to pid 9506
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 13 done, 7 running, 193 todo
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 job_task z8ta6-ot0gb-blcloqm3400c4bg
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 child 13267 started on humgen-05-15.6
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 stderr srun: error: Unable to create job step: Job/step already completing or completed
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970  backing off node humgen-05-15 for 60 seconds
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-zex553q8rtr26xg-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zex553q8rtr26xg-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=19 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-zex553q8rtr26xg --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a6f17cb73cb3c915242e59a4820b846d9f1d5639e79f087b4e613e93f360271c/cgroup.procs
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/a6f17cb73cb3c915242e59a4820b846d9f1d5639e79f087b4e613e93f360271c/memory.stat
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5861376 rss
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a6f17cb73cb3c915242e59a4820b846d9f1d5639e79f087b4e613e93f360271c/cpuacct.stat
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/a6f17cb73cb3c915242e59a4820b846d9f1d5639e79f087b4e613e93f360271c/cpuset.cpus
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: cpu 0.0300 user 0.0200 sys 32 cpus
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/a6f17cb73cb3c915242e59a4820b846d9f1d5639e79f087b4e613e93f360271c/blkio.io_service_bytes
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Mounting task input collection to get interval_list
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Interval_List collection mounted at /keep/76a230c91cc5d0053b75306d58906b2a+94
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Total chunk length is 15498872
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Splitting chunk into 1 intervals of size ~15498872
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Have 1 intervals
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:32:25 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.22.13267','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-blcloqm3400c4bg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-blcloqm3400c4bg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-blcloqm3400c4bg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=22 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-blcloqm3400c4bg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:32:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 child 9506 on humgen-05-15.2 exit 1 success=
2016-12-07_01:32:26 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-12-07_01:32:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 failure (#1, temporary) after 111 seconds
2016-12-07_01:32:27 z8ta6-8i9sb-r52unvxj6avgrio 4970 10 task output (0 bytes): 
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 13 done, 7 running, 193 todo
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Have 5400 potential reusable task outputs
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Have 5400 potentially reusable tasks
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Creating task to process interval: [10:130336010-135534747 11:1-10300134]
2016-12-07_01:32:28 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr No reusable JobTask matched key parameters [u'117_of_200', u'a986cd76d508917b0b96bd60c6107961+14704', '10:130336010-135534747 11:1-10300134', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr Ending task 1 successfully
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 stderr crunchstat: blkio:0:0 0 write 12567 read -- interval 10.0000 seconds 0 write 12567 read
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 child 12499 on humgen-05-15.8 exit 0 success=true
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 success in 27 seconds
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970 19 task output (0 bytes): 
2016-12-07_01:32:29 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 14 done, 6 running, 194 todo
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 job_task z8ta6-ot0gb-cqpt669hl2r0t18
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 child 13584 started on humgen-05-15.8
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 job_task z8ta6-ot0gb-xo8sk9whc1n6eon
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 child 13597 started on humgen-05-15.2
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.23.13584','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-cqpt669hl2r0t18-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-cqpt669hl2r0t18-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-cqpt669hl2r0t18-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=23 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-cqpt669hl2r0t18 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.24.13597','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xo8sk9whc1n6eon-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xo8sk9whc1n6eon-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xo8sk9whc1n6eon-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.2\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=24 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=2 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-xo8sk9whc1n6eon --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.2 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.2" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr srun: Job step created
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-mrupikb9g03ehny-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-mrupikb9g03ehny-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=16 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-mrupikb9g03ehny --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:32:36 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/39cd517fb1603f4c8689d396091cf837b3dfed416be85408e8308df8980cdc04/cgroup.procs
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/39cd517fb1603f4c8689d396091cf837b3dfed416be85408e8308df8980cdc04/memory.stat
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: mem 86016 cache 0 swap 0 pgmajfault 5816320 rss
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/39cd517fb1603f4c8689d396091cf837b3dfed416be85408e8308df8980cdc04/cpuacct.stat
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/39cd517fb1603f4c8689d396091cf837b3dfed416be85408e8308df8980cdc04/cpuset.cpus
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: cpu 0.0500 user 0.0200 sys 32 cpus
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/39cd517fb1603f4c8689d396091cf837b3dfed416be85408e8308df8980cdc04/blkio.io_service_bytes
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Mounting task input collection to get interval_list
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Interval_List collection mounted at /keep/8b0a6de6be0a7b3dd7360336477e02fd+94
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Total chunk length is 15618872
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Have 1 intervals
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:32:37 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 14 done, 8 running, 192 todo
2016-12-07_01:32:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:32:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:32:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:32:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:32:39 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:32:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24092672 rss
2016-12-07_01:32:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: cpu 0.5000 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.4500 user 0.1400 sys
2016-12-07_01:32:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr crunchstat: net:eth0 21877 tx 238414 rx -- interval 10.0001 seconds 21787 tx 238324 rx
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Have 5400 potential reusable task outputs
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Have 5400 potentially reusable tasks
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Creating task to process interval: [10:83479394-99098265]
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr No reusable JobTask matched key parameters [u'114_of_200', u'c3bc3b07edcf5d2b7b9bfb2b5154d56f+14700', '10:83479394-99098265', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:32:42 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 stderr Ending task 1 successfully
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr srun: Job step created
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 child 11540 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 success in 63 seconds
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 16 task output (0 bytes): 
2016-12-07_01:32:43 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 15 done, 7 running, 193 todo
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 job_task z8ta6-ot0gb-ltvbgm6zwzhs2or
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 child 13988 started on humgen-05-15.7
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ljma8zh3j9drwfa-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ljma8zh3j9drwfa-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=18 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ljma8zh3j9drwfa --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8c4abf0342850e2dc3f24beb2757f94765b7f1659ec0cc5fa0387d0493e7a175/cgroup.procs
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8c4abf0342850e2dc3f24beb2757f94765b7f1659ec0cc5fa0387d0493e7a175/memory.stat
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 4964352 rss
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8c4abf0342850e2dc3f24beb2757f94765b7f1659ec0cc5fa0387d0493e7a175/cpuacct.stat
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8c4abf0342850e2dc3f24beb2757f94765b7f1659ec0cc5fa0387d0493e7a175/cpuset.cpus
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: cpu 0.0300 user 0.0000 sys 32 cpus
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8c4abf0342850e2dc3f24beb2757f94765b7f1659ec0cc5fa0387d0493e7a175/blkio.io_service_bytes
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Mounting task input collection to get interval_list
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Interval_List collection mounted at /keep/359a96b521b56e8d8e36b57450fa80ac+94
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Total chunk length is 15618872
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Have 1 intervals
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.25.13988','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ltvbgm6zwzhs2or-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ltvbgm6zwzhs2or-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ltvbgm6zwzhs2or-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.7\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=25 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=7 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-ltvbgm6zwzhs2or --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:32:50 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 15 done, 8 running, 192 todo
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Have 5400 potential reusable task outputs
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Have 5400 potentially reusable tasks
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Creating task to process interval: [10:114717138-130336009]
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr No reusable JobTask matched key parameters [u'116_of_200', u'6365525c0ee094fee362794dbbab6a30+14700', '10:114717138-130336009', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 stderr Ending task 1 successfully
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 child 12495 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 success in 51 seconds
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970 18 task output (0 bytes): 
2016-12-07_01:32:53 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 notice: task is not in slurm queue but srun process 12176 has not exited
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 16 done, 7 running, 193 todo
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 job_task z8ta6-ot0gb-dym505mzgyoooiy
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 child 14344 started on humgen-05-15.4
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-cqpt669hl2r0t18-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-cqpt669hl2r0t18-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=23 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-cqpt669hl2r0t18 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/08e070f837ecb33016e9631a0231e6406feabb2decf1217c0255d1bd4146fc40/cgroup.procs
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/08e070f837ecb33016e9631a0231e6406feabb2decf1217c0255d1bd4146fc40/memory.stat
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: mem 94208 cache 0 swap 0 pgmajfault 5206016 rss
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/08e070f837ecb33016e9631a0231e6406feabb2decf1217c0255d1bd4146fc40/cpuacct.stat
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/08e070f837ecb33016e9631a0231e6406feabb2decf1217c0255d1bd4146fc40/cpuset.cpus
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/08e070f837ecb33016e9631a0231e6406feabb2decf1217c0255d1bd4146fc40/blkio.io_service_bytes
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr crunchstat: net:eth0 90 tx 90 rx
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Interval_List collection mounted at /keep/b34c79e916b3f4868c677e831b11c26d+94
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Total chunk length is 15618872
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Have 1 intervals
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:00 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.26.14344','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-dym505mzgyoooiy-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-dym505mzgyoooiy-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-dym505mzgyoooiy-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=26 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-dym505mzgyoooiy --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:01 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:33:01 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 16 done, 8 running, 192 todo
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Creating task to process interval: [11:41537879-57156750]
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr No reusable JobTask matched key parameters [u'120_of_200', u'4253ced78675bc039ec63c878e7f8d99+14700', '11:41537879-57156750', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 stderr Ending task 1 successfully
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr srun: Job step created
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 child 13584 on humgen-05-15.8 exit 0 success=true
2016-12-07_01:33:03 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 success in 27 seconds
2016-12-07_01:33:04 z8ta6-8i9sb-r52unvxj6avgrio 4970 23 task output (0 bytes): 
2016-12-07_01:33:04 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 17 done, 7 running, 193 todo
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 job_task z8ta6-ot0gb-62gm4qd6vfzcrvk
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 child 14644 started on humgen-05-15.8
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.27.14644','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-62gm4qd6vfzcrvk-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-62gm4qd6vfzcrvk-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-62gm4qd6vfzcrvk-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.8\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=27 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=8 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-62gm4qd6vfzcrvk --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-9pmmsjzn1rgx4qz-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-9pmmsjzn1rgx4qz-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.3.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=20 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.3 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=3 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.3 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-9pmmsjzn1rgx4qz --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/050c1a5d83dfde73d79158350973e60d3e2c9ae2b8b1ba58ccdea162309a9d1b/cgroup.procs
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/050c1a5d83dfde73d79158350973e60d3e2c9ae2b8b1ba58ccdea162309a9d1b/memory.stat
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: mem 16384 cache 0 swap 0 pgmajfault 4317184 rss
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/050c1a5d83dfde73d79158350973e60d3e2c9ae2b8b1ba58ccdea162309a9d1b/cpuacct.stat
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/050c1a5d83dfde73d79158350973e60d3e2c9ae2b8b1ba58ccdea162309a9d1b/cpuset.cpus
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: cpu 0.0500 user 0.0000 sys 32 cpus
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/050c1a5d83dfde73d79158350973e60d3e2c9ae2b8b1ba58ccdea162309a9d1b/blkio.io_service_bytes
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr crunchstat: net:eth0 0 tx 0 rx
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Interval_List collection mounted at /keep/d9aa702cd7b3a1e6b6dec54b4a3d8e35+94
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Total chunk length is 15618872
2016-12-07_01:33:10 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Have 1 intervals
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:11 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 17 done, 8 running, 192 todo
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Creating task to process interval: [11:10300135-25919006]
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr No reusable JobTask matched key parameters [u'118_of_200', u'79b22e2a5c2a107fd090c78ed23ff2e7+14700', '11:10300135-25919006', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 stderr Ending task 1 successfully
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 child 12503 on humgen-05-15.3 exit 0 success=true
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 success in 71 seconds
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970 20 task output (0 bytes): 
2016-12-07_01:33:13 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 notice: task is not in slurm queue but srun process 12866 has not exited
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 18 done, 7 running, 193 todo
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 job_task z8ta6-ot0gb-f5vx8p9vrdy4mpx
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 child 14959 started on humgen-05-15.3
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-dym505mzgyoooiy-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-dym505mzgyoooiy-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.4.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=26 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.4 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=4 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.4 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-dym505mzgyoooiy --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8382eb0899e8d62d641e6b1f7daa227354802665fc0759355b6ed95fbfe80d3d/cgroup.procs
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/8382eb0899e8d62d641e6b1f7daa227354802665fc0759355b6ed95fbfe80d3d/memory.stat
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5750784 rss
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/8382eb0899e8d62d641e6b1f7daa227354802665fc0759355b6ed95fbfe80d3d/cpuacct.stat
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/8382eb0899e8d62d641e6b1f7daa227354802665fc0759355b6ed95fbfe80d3d/cpuset.cpus
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/8382eb0899e8d62d641e6b1f7daa227354802665fc0759355b6ed95fbfe80d3d/blkio.io_service_bytes
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Interval_List collection mounted at /keep/ddb0ad35fd81534c2371a454b4e54268+94
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Total chunk length is 15618872
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Have 1 intervals
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:20 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.28.14959','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-f5vx8p9vrdy4mpx-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-f5vx8p9vrdy4mpx-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-f5vx8p9vrdy4mpx-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.3\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=28 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=3 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-f5vx8p9vrdy4mpx --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.3 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.3" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:21 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 18 done, 8 running, 192 todo
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Creating task to process interval: [11:88394495-104013366]
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr No reusable JobTask matched key parameters [u'123_of_200', u'b28694acffb44b65c84fe09c422e709c+14700', '11:88394495-104013366', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:22 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 stderr Ending task 1 successfully
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr srun: Job step created
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 child 14344 on humgen-05-15.4 exit 0 success=true
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 success in 23 seconds
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970 26 task output (0 bytes): 
2016-12-07_01:33:23 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:29 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 19 done, 7 running, 193 todo
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 job_task z8ta6-ot0gb-qdl6sjf1fvebcfg
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 child 15286 started on humgen-05-15.4
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-xn87jpqo3q9dql1-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-xn87jpqo3q9dql1-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.1.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=17 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.1 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=1 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.1 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-xn87jpqo3q9dql1 --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a670ae0006d74e66a06ca3f2f04e120dde4b7712a5ef861be87fd506df7d4b02/cgroup.procs
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/a670ae0006d74e66a06ca3f2f04e120dde4b7712a5ef861be87fd506df7d4b02/memory.stat
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5836800 rss
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/a670ae0006d74e66a06ca3f2f04e120dde4b7712a5ef861be87fd506df7d4b02/cpuacct.stat
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/a670ae0006d74e66a06ca3f2f04e120dde4b7712a5ef861be87fd506df7d4b02/cpuset.cpus
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/a670ae0006d74e66a06ca3f2f04e120dde4b7712a5ef861be87fd506df7d4b02/blkio.io_service_bytes
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Interval_List collection mounted at /keep/e05c9a56f87bbb0e87bd78edfb2138bb+94
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Total chunk length is 15618872
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Have 1 intervals
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.29.15286','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qdl6sjf1fvebcfg-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-qdl6sjf1fvebcfg-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-qdl6sjf1fvebcfg-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.4\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=29 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=4 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-qdl6sjf1fvebcfg --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.4 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.4" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:30 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 19 done, 8 running, 192 todo
2016-12-07_01:33:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:32 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Creating task to process interval: [10:99098266-114717137]
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr No reusable JobTask matched key parameters [u'115_of_200', u'5601c60d4022448c1a998597fc7d32dc+14700', '10:99098266-114717137', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 stderr Ending task 1 successfully
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr srun: Job step created
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 28 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 child 12176 on humgen-05-15.1 exit 0 success=true
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 success in 103 seconds
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970 17 task output (0 bytes): 
2016-12-07_01:33:33 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 notice: task is not in slurm queue but srun process 13597 has not exited
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 notice: task is not in slurm queue but srun process 13267 has not exited
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 20 done, 7 running, 193 todo
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 job_task z8ta6-ot0gb-4ojxt1g1nb8k256
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 child 15891 started on humgen-05-15.1
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.30.15891','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-4ojxt1g1nb8k256-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-4ojxt1g1nb8k256-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-4ojxt1g1nb8k256-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.1\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=30 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=1 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-4ojxt1g1nb8k256 --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.1 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.1" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-zi4wk7zwhebjuoo-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-zi4wk7zwhebjuoo-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.5.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=21 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.5 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=5 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.5 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-zi4wk7zwhebjuoo --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/31b277122a6ade22b034c2ebeba4c93d4078354afe2abb7dd34db292db4d5a6d/cgroup.procs
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/31b277122a6ade22b034c2ebeba4c93d4078354afe2abb7dd34db292db4d5a6d/memory.stat
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5791744 rss
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/31b277122a6ade22b034c2ebeba4c93d4078354afe2abb7dd34db292db4d5a6d/cpuacct.stat
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/31b277122a6ade22b034c2ebeba4c93d4078354afe2abb7dd34db292db4d5a6d/cpuset.cpus
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: cpu 0.0400 user 0.0100 sys 32 cpus
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/31b277122a6ade22b034c2ebeba4c93d4078354afe2abb7dd34db292db4d5a6d/blkio.io_service_bytes
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: net:eth0 0 tx 0 rx
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Interval_List collection mounted at /keep/dbba1077cec452fe00018827674a6082+94
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Total chunk length is 15618872
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Have 1 intervals
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:40 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:41 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 20 done, 8 running, 192 todo
2016-12-07_01:33:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:33:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:33:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:33:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:33:43 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: blkio:0:0 0 write 12526 read -- interval 10.0000 seconds 0 write 12526 read
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24293376 rss
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: cpu 0.5300 user 0.1500 sys 32 cpus -- interval 10.0002 seconds 0.4900 user 0.1400 sys
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr crunchstat: net:eth0 26497 tx 3078455 rx -- interval 10.0002 seconds 26497 tx 3078455 rx
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Creating task to process interval: [11:25919007-41537878]
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr No reusable JobTask matched key parameters [u'119_of_200', u'e715710b28adf5479c2b4ee6a270292a+14700', '11:25919007-41537878', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:44 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 stderr Ending task 1 successfully
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 29 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr srun: Job step created
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 child 12866 on humgen-05-15.5 exit 0 success=true
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 success in 91 seconds
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970 21 task output (0 bytes): 
2016-12-07_01:33:45 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 21 done, 7 running, 193 todo
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 job_task z8ta6-ot0gb-d7p9u5ff3wkrs4o
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 child 16264 started on humgen-05-15.5
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 31 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.31.16264','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-d7p9u5ff3wkrs4o-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-d7p9u5ff3wkrs4o-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-d7p9u5ff3wkrs4o-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.5\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=31 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=5 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-d7p9u5ff3wkrs4o --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.5 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.5" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-blcloqm3400c4bg-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-blcloqm3400c4bg-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.6.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=22 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.6 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=6 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.6 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-blcloqm3400c4bg --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d9561a01c7a8cb0caf09bc5fb6c9c61c13539d9f43619c1cde9e9bb8ac354132/cgroup.procs
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d9561a01c7a8cb0caf09bc5fb6c9c61c13539d9f43619c1cde9e9bb8ac354132/memory.stat
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 6094848 rss
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d9561a01c7a8cb0caf09bc5fb6c9c61c13539d9f43619c1cde9e9bb8ac354132/cpuacct.stat
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d9561a01c7a8cb0caf09bc5fb6c9c61c13539d9f43619c1cde9e9bb8ac354132/cpuset.cpus
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: cpu 0.0400 user 0.0000 sys 32 cpus
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d9561a01c7a8cb0caf09bc5fb6c9c61c13539d9f43619c1cde9e9bb8ac354132/blkio.io_service_bytes
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Mounting task input collection to get interval_list
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Interval_List collection mounted at /keep/a19791935337560ca00961b9d0c50277+92
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Total chunk length is 15618872
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Have 1 intervals
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:33:52 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 21 done, 8 running, 192 todo
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 notice: task is not in slurm queue but srun process 13988 has not exited
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: keepcalls 0 put 1 get -- interval 10.0000 seconds 0 put 1 get
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: net:keep0 0 tx 2509051 rx -- interval 10.0000 seconds 0 tx 2509051 rx
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: keepcache 0 hit 1 miss -- interval 10.0000 seconds 0 hit 1 miss
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: fuseops 0 write 1 read -- interval 10.0000 seconds 0 write 1 read
2016-12-07_01:33:55 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: blkio:0:0 0 write 12525 read -- interval 10.0000 seconds 0 write 12525 read
2016-12-07_01:33:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: mem 962560 cache 0 swap 0 pgmajfault 24088576 rss
2016-12-07_01:33:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: cpu 0.4900 user 0.1600 sys 32 cpus -- interval 10.0002 seconds 0.4500 user 0.1600 sys
2016-12-07_01:33:56 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr crunchstat: net:eth0 22183 tx 238403 rx
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Have 5400 potential reusable task outputs
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Have 5400 potentially reusable tasks
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Creating task to process interval: [1:156188721-171807592]
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr No reusable JobTask matched key parameters [u'11_of_200', u'0407d47186859c17195bd2efb49797bb+14616', '1:156188721-171807592', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:33:57 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 stderr Ending task 1 successfully
2016-12-07_01:33:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 child 13267 on humgen-05-15.6 exit 0 success=true
2016-12-07_01:33:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 success in 93 seconds
2016-12-07_01:33:58 z8ta6-8i9sb-r52unvxj6avgrio 4970 22 task output (0 bytes): 
2016-12-07_01:33:58 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:34:04 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 22 done, 7 running, 193 todo
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 job_task z8ta6-ot0gb-0y41phqbnenlsfs
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 child 16643 started on humgen-05-15.6
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 30 stderr srun: Job step creation temporarily disabled, retrying
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 32 stderr starting: ['srun','--nodelist=humgen-05-15','-n1','-c1','-N1','-D','/data/crunch-tmp','--job-name=z8ta6-8i9sb-r52unvxj6avgrio.32.16643','bash','-c','if [ -e \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 ]; then rm -rf \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6; fi; mkdir -p \\/data\\/crunch\\-tmp\\/crunch\\-job \\/data\\/crunch\\-tmp\\/crunch\\-job\\/work \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep && cd \\/data\\/crunch\\-tmp\\/crunch\\-job && MEM=$(awk \'($1 == "MemTotal:"){print $2}\' </proc/meminfo) && SWAP=$(awk \'($1 == "SwapTotal:"){print $2}\' </proc/meminfo) && MEMLIMIT=$(( ($MEM * 95) / (8 * 100) )) && let SWAPLIMIT=$MEMLIMIT+$SWAP && declare -a VOLUMES=() && if which crunchrunner >/dev/null ; then VOLUMES+=("--volume=$(which crunchrunner):/usr/local/bin/crunchrunner") ; fi && if test -f /etc/ssl/certs/ca-certificates.crt ; then VOLUMES+=("--volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt") ; elif test -f /etc/pki/tls/certs/ca-bundle.crt ; then VOLUMES+=("--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/arvados/ca-certificates.crt") ; fi && exec arv-mount --read-write --mount-by-pdh=by_pdh --mount-tmp=tmp --crunchstat-interval=10 --allow-other --file-cache=20447232000 \\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep --exec crunchstat -cgroup-root=\\/sys\\/fs\\/cgroup -cgroup-parent=docker -cgroup-cid=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0y41phqbnenlsfs-0.cid -poll=10000 /usr/bin/docker run  --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-0y41phqbnenlsfs-0 --attach=stdout --attach=stderr --attach=stdin -i  --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-0y41phqbnenlsfs-0.cid --sig-proxy --memory=${MEMLIMIT}k --memory-swap=${SWAPLIMIT}k --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/by_pdh\\:\\/keep\\:ro --volume=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/task\\/humgen\\-05\\-15\\.6\\.keep\\/tmp\\:\\/keep_tmp --volume=/tmp "${VOLUMES[@]}" --env=CRUNCH_JOB_DOCKER_RUN_ARGS\\=\\ \\-\\-add\\-host\\=api\\.arvados\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.10\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.11\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-01\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.12\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.13\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.14\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-02\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.15\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.16\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.17\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-03\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.18\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-01\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.19\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-02\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.20\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\:172\\.17\\.180\\.21\\ \\-\\-add\\-host\\=humgen\\-04\\-03\\.internal\\.sanger\\.ac\\.uk\\:172\\.17\\.180\\.21 --env=TASK_SEQUENCE\\=1 --env=TASK_KEEPMOUNT\\=\\/keep --env=JOB_PARAMETER_INPUTS_COLLECTION\\=b0dd82bb014b6ab8e64968ebdc07aad9\\+2566847 --env=CRUNCH_SRC_COMMIT\\=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE\\=32 --env=CRUNCH_INSTALL\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/opt --env=CRUNCH_GIT_ARCHIVE_HASH\\=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER\\=\\/tmp\\/crunch_refresh_trigger --env=ARVADOS_API_TOKEN\\=[...] --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION\\=4a72f79fd2ed51ac692bc326fb168f37\\+10043 --env=CRUNCH_WORK\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/work --env=CRUNCH_TMP\\=\\/data\\/crunch\\-tmp\\/crunch\\-job --env=TASK_TMPDIR\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=CRUNCH_JOB_UUID\\=z8ta6\\-8i9sb\\-r52unvxj6avgrio --env=TASK_SLOT_NUMBER\\=6 --env=CRUNCH_SRC_URL\\=\\/var\\/lib\\/arvados\\/internal\\.git --env=TASK_SLOT_NODE\\=humgen\\-05\\-15 --env=JOB_SCRIPT\\=gatk\\-combinegvcfs\\.py --env=CRUNCH_NODE_SLOTS\\=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION\\=1b447d1aa006cde57df03042736750b9\\+7894 --env=CRUNCH_JOB_DOCKER_BIN\\=\\/usr\\/bin\\/docker --env=TASK_WORK\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 --env=TASK_KEEPMOUNT_TMP\\=\\/keep_tmp --env=ARVADOS_API_HOST\\=api\\.arvados\\.sanger\\.ac\\.uk --env=JOB_WORK\\=\\/tmp\\/crunch\\-job\\-work --env=TASK_UUID\\=z8ta6\\-ot0gb\\-0y41phqbnenlsfs --env=CRUNCH_SRC\\=\\/data\\/crunch\\-tmp\\/crunch\\-job\\/src --env=HOME\\=\\/tmp\\/crunch\\-job\\-task\\-work\\/humgen\\-05\\-15\\.6 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c \'python -c "from pkg_resources import get_distribution as get; print \\"Using Arvados SDK version\\", get(\\"arvados-python-client\\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.6" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; else   exec \\/data\\/crunch\\-tmp\\/crunch\\-job\\/src\\/crunch_scripts\\/gatk\\-combinegvcfs\\.py ; fi\'']
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr srun: Job step created
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-ltvbgm6zwzhs2or-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-ltvbgm6zwzhs2or-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.7.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=25 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.7 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=7 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.7 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-ltvbgm6zwzhs2or --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.7 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.7" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/3930a4fc92a309581fd68e642433f87555ceae849952a82bbe84201f576d0012/cgroup.procs
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/3930a4fc92a309581fd68e642433f87555ceae849952a82bbe84201f576d0012/memory.stat
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5177344 rss
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/3930a4fc92a309581fd68e642433f87555ceae849952a82bbe84201f576d0012/cpuacct.stat
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/3930a4fc92a309581fd68e642433f87555ceae849952a82bbe84201f576d0012/cpuset.cpus
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: cpu 0.0300 user 0.0100 sys 32 cpus
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/3930a4fc92a309581fd68e642433f87555ceae849952a82bbe84201f576d0012/blkio.io_service_bytes
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Mounting task input collection to get interval_list
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Interval_List collection mounted at /keep/bbfdffd38f1432cef552ced110ab3a90+94
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Total chunk length is 15618872
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Have 1 intervals
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:34:05 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 22 done, 8 running, 192 todo
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Have 5400 potential reusable task outputs
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Batch had 5400 items, 5400 items retrieved so far (batch_offset at 0 out of 5400)
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr All batches complete, retrieved 5400 items in total
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Have 5400 potentially reusable tasks
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Creating task to process interval: [11:72775623-88394494]
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr No reusable JobTask matched key parameters [u'122_of_200', u'af62e8edc576e4baf02413c2134c9270+14700', '11:72775623-88394494', u'd527a0b11143ebf18be6c52ff6c09552+2163']
2016-12-07_01:34:07 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 stderr Ending task 1 successfully
2016-12-07_01:34:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 child 13988 on humgen-05-15.7 exit 0 success=true
2016-12-07_01:34:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 success in 78 seconds
2016-12-07_01:34:08 z8ta6-8i9sb-r52unvxj6avgrio 4970 25 task output (0 bytes): 
2016-12-07_01:34:08 z8ta6-8i9sb-r52unvxj6avgrio 4970  1 tasks exited (1 succeeded), checking for new tasks from API server.
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970  Got 1 new tasks from API server.
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 killing orphaned srun process 13597 (task not in slurm queue, no stderr received in last 74s)
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 sending 2x signal 2 to pid 13597
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970  status: 23 done, 7 running, 193 todo
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 job_task z8ta6-ot0gb-r9pvowy0tacjusv
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 33 child 16999 started on humgen-05-15.7
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr srun: Job step created
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: Running [/usr/bin/docker run --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --name=z8ta6-ot0gb-62gm4qd6vfzcrvk-0 --attach=stdout --attach=stderr --attach=stdin -i --cidfile=/data/crunch-tmp/crunch-job/z8ta6-ot0gb-62gm4qd6vfzcrvk-0.cid --sig-proxy --memory=31356661k --memory-swap=64911089k --volume=/data/crunch-tmp/crunch-job/src:/data/crunch-tmp/crunch-job/src:ro --volume=/data/crunch-tmp/crunch-job/opt:/data/crunch-tmp/crunch-job/opt:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/by_pdh:/keep:ro --volume=/data/crunch-tmp/crunch-job/task/humgen-05-15.8.keep/tmp:/keep_tmp --volume=/tmp --volume=/etc/ssl/certs/ca-certificates.crt:/etc/arvados/ca-certificates.crt --env=CRUNCH_JOB_DOCKER_RUN_ARGS= --add-host=api.arvados.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-01:172.17.180.10 --add-host=humgen-01-01.internal.sanger.ac.uk:172.17.180.10 --add-host=humgen-01-02:172.17.180.11 --add-host=humgen-01-02.internal.sanger.ac.uk:172.17.180.11 --add-host=humgen-01-03:172.17.180.12 --add-host=humgen-01-03.internal.sanger.ac.uk:172.17.180.12 --add-host=humgen-02-01:172.17.180.13 --add-host=humgen-02-01.internal.sanger.ac.uk:172.17.180.13 --add-host=humgen-02-02:172.17.180.14 --add-host=humgen-02-02.internal.sanger.ac.uk:172.17.180.14 --add-host=humgen-02-03:172.17.180.15 --add-host=humgen-02-03.internal.sanger.ac.uk:172.17.180.15 --add-host=humgen-03-01:172.17.180.16 --add-host=humgen-03-01.internal.sanger.ac.uk:172.17.180.16 --add-host=humgen-03-02:172.17.180.17 --add-host=humgen-03-02.internal.sanger.ac.uk:172.17.180.17 --add-host=humgen-03-03:172.17.180.18 --add-host=humgen-03-03.internal.sanger.ac.uk:172.17.180.18 --add-host=humgen-04-01:172.17.180.19 --add-host=humgen-04-01.internal.sanger.ac.uk:172.17.180.19 --add-host=humgen-04-02:172.17.180.20 --add-host=humgen-04-02.internal.sanger.ac.uk:172.17.180.20 --add-host=humgen-04-03:172.17.180.21 --add-host=humgen-04-03.internal.sanger.ac.uk:172.17.180.21 --env=TASK_SEQUENCE=1 --env=TASK_KEEPMOUNT=/keep --env=JOB_PARAMETER_INPUTS_COLLECTION=b0dd82bb014b6ab8e64968ebdc07aad9+2566847 --env=CRUNCH_SRC_COMMIT=794d64bc0ceb8bd4112397fd63bb3f97ab67e2b4 --env=TASK_QSEQUENCE=27 --env=CRUNCH_INSTALL=/data/crunch-tmp/crunch-job/opt --env=CRUNCH_GIT_ARCHIVE_HASH=4b8cf8e1274f8d418cb494d3ea978152 --env=CRUNCH_REFRESH_TRIGGER=/tmp/crunch_refresh_trigger --env=ARVADOS_API_TOKEN=27sq3s7iygo9ifgtr53gpxyxguijeit34v5x2sxx94uuezdldb --env=JOB_PARAMETER_INTERVAL_LISTS_COLLECTION=4a72f79fd2ed51ac692bc326fb168f37+10043 --env=CRUNCH_WORK=/data/crunch-tmp/crunch-job/work --env=CRUNCH_TMP=/data/crunch-tmp/crunch-job --env=TASK_TMPDIR=/tmp/crunch-job-task-work/humgen-05-15.8 --env=JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=CRUNCH_JOB_UUID=z8ta6-8i9sb-r52unvxj6avgrio --env=TASK_SLOT_NUMBER=8 --env=CRUNCH_SRC_URL=/var/lib/arvados/internal.git --env=TASK_SLOT_NODE=humgen-05-15 --env=JOB_SCRIPT=gatk-combinegvcfs.py --env=CRUNCH_NODE_SLOTS=8 --env=JOB_PARAMETER_REFERENCE_COLLECTION=1b447d1aa006cde57df03042736750b9+7894 --env=CRUNCH_JOB_DOCKER_BIN=/usr/bin/docker --env=TASK_WORK=/tmp/crunch-job-task-work/humgen-05-15.8 --env=TASK_KEEPMOUNT_TMP=/keep_tmp --env=ARVADOS_API_HOST=api.arvados.sanger.ac.uk --env=JOB_WORK=/tmp/crunch-job-work --env=TASK_UUID=z8ta6-ot0gb-62gm4qd6vfzcrvk --env=CRUNCH_SRC=/data/crunch-tmp/crunch-job/src --env=HOME=/tmp/crunch-job-task-work/humgen-05-15.8 571ee2c9c7d35848cae8f12f458b338dc49d94fcdfc69349da6198030aff18f3 /bin/sh -c python -c "from pkg_resources import get_distribution as get; print \"Using Arvados SDK version\", get(\"arvados-python-client\").version">&2 2>/dev/null; mkdir -p "/tmp/crunch-job-work" "/tmp/crunch-job-task-work/humgen-05-15.8" && if which stdbuf >/dev/null ; then   exec  stdbuf --output=0 --error=0  \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; else   exec \/data\/crunch\-tmp\/crunch\-job\/src\/crunch_scripts\/gatk\-combinegvcfs\.py ; fi]
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d040156217c1762059da8b84f1f81b101ecbbd81e4c074304dee45188a8adb73/cgroup.procs
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/memory/docker/d040156217c1762059da8b84f1f81b101ecbbd81e4c074304dee45188a8adb73/memory.stat
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: mem 0 cache 0 swap 0 pgmajfault 5271552 rss
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuacct/docker/d040156217c1762059da8b84f1f81b101ecbbd81e4c074304dee45188a8adb73/cpuacct.stat
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/d040156217c1762059da8b84f1f81b101ecbbd81e4c074304dee45188a8adb73/cpuset.cpus
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: cpu 0.0200 user 0.0100 sys 32 cpus
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/blkio/docker/d040156217c1762059da8b84f1f81b101ecbbd81e4c074304dee45188a8adb73/blkio.io_service_bytes
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Using Arvados SDK version 0.1.20160727172639
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Mounting task input collection to get interval_list
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Interval_List collection mounted at /keep/b4389106228a3052a1940c08eb93c60e+94
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Total chunk length is 15618872
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Splitting chunk into 1 intervals of size ~15618872
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Have 1 intervals
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Querying API server for jobs matching filters [["script", "=", "gatk-combinegvcfs.py"], ["repository", "=", "jr17/hgi"], ["script_version", "in git", "1f6e1e0b8bb12c573dd253d7900ef55305d55aa1"], ["docker_image_locator", "in docker", "9289c4f1fce1d1dedf9cc7876c33f936+671"]]
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Found 163 similar previous jobs, checking them for reusable tasks
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr Batch had 163 items, 163 items retrieved so far (batch_offset at 0 out of 163)
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 27 stderr All batches complete, retrieved 163 items in total
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 stderr srun: Cancelled pending job step with signal 2
2016-12-07_01:34:15 z8ta6-8i9sb-r52unvxj6avgrio 4970 24 stderr srun: error: Unable to cre