Bug #8284: [Crunch] Task killed even though it is still running - Arvados

Actions

Copy link

Bug #8284

closed

[Crunch] Task killed even though it is still running

Added by Peter Amstutz over 8 years ago. Updated about 8 years ago.

Status:

Resolved

Priority:

Normal

Assigned To:

Tom Clegg

Category:

Crunch

Target version:

2016-02-17 Sprint

Story points:

0.5

Description

Original log https://cloud.curoverse.com/collections/download/qr1hi-4zz18-oiq4t3dbe66s1ms/1n0ptseu5qsdxix1vyevhu0tcxiouel5ty76316hdeu7gsa334/z8ta6-8i9sb-xb0qkz4qvfm5h8j.log.txt

2016-01-22_13:10:36 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr WARNING: Your kernel does not support swap limit capabilities, memory limited without swap.
2016-01-22_13:10:36 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: notice: stats not available: stat memory.stat, statgroup memory, cid b99b87e6101571d37236efeda702735350e3cadcd6daa1ab00b8672e48eb23a6, parent docker, root /sys/fs/cgroup
2016-01-22_13:10:36 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: notice: stats not available: stat cpuacct.stat, statgroup cpuacct, cid b99b87e6101571d37236efeda702735350e3cadcd6daa1ab00b8672e48eb23a6, parent docker, root /sys/fs/cgroup
2016-01-22_13:10:36 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: notice: stats not available: stat blkio.io_service_bytes, statgroup blkio, cid b99b87e6101571d37236efeda702735350e3cadcd6daa1ab00b8672e48eb23a6, parent docker, root /sys/fs/cgroup
2016-01-22_13:10:36 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: notice: stats not available: stat cgroup.procs, statgroup cpuacct, cid b99b87e6101571d37236efeda702735350e3cadcd6daa1ab00b8672e48eb23a6, parent docker, root /sys/fs/cgroup
2016-01-22_13:10:36 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Using Arvados SDK version 0.1.20151215145437
2016-01-22_13:10:36 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Mounting reference FASTA collection
2016-01-22_13:10:36 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Successfully read 1024 bytes from /keep/9a15a1d495a6efa0f3e05f1e851b694e+2227/hs38DH.fa
2016-01-22_13:10:36 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcalls 0 put 3 get -- interval 10.0000 seconds 0 put 3 get
2016-01-22_13:10:36 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:keep0 0 tx 67108864 rx -- interval 10.0000 seconds 0 tx 67108864 rx

2016-01-22_13:10:59 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 notice: task is not in slurm queue but srun process 31346 has not exited

2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcache 0 hit 3 miss -- interval 10.0000 seconds 0 hit 3 miss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: fuseops 0 write 2 read -- interval 10.0000 seconds 0 write 2 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: blkio:0:0 0 write 16384 read -- interval 10.0000 seconds 0 write 16384 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Successfully read 1024 bytes from /keep/9a15a1d495a6efa0f3e05f1e851b694e+2227/hs38DH.fa.fai
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Mounting chunk collection
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Successfully read 22 bytes from /keep/1bfc3b9a382ce0ab14c8e156745133c2+128+A55d05af2acd1fd610ea247d1015d61d10eda03e6@56b49aec/hs38DH.dict.32_of_400.region_list
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Mounting task input collection
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: notice: stats moved from  to /sys/fs/cgroup/memory/docker/b99b87e6101571d37236efeda702735350e3cadcd6daa1ab00b8672e48eb23a6/memory.stat
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: mem 512000 cache 0 pgmajfault 18186240 rss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: notice: stats moved from  to /sys/fs/cgroup/cpuacct/docker/b99b87e6101571d37236efeda702735350e3cadcd6daa1ab00b8672e48eb23a6/cpuacct.stat
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: notice: reading stats from /sys/fs/cgroup/cpuset/docker/b99b87e6101571d37236efeda702735350e3cadcd6daa1ab00b8672e48eb23a6/cpuset.cpus
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: cpu 0.2100 user 0.1600 sys 40 cpus
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: notice: stats moved from  to /sys/fs/cgroup/blkio/docker/b99b87e6101571d37236efeda702735350e3cadcd6daa1ab00b8672e48eb23a6/blkio.io_service_bytes
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: notice: stats moved from  to /sys/fs/cgroup/cpuacct/docker/b99b87e6101571d37236efeda702735350e3cadcd6daa1ab00b8672e48eb23a6/cgroup.procs
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:eth0 4618 tx 184787 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcalls 0 put 6 get -- interval 10.0000 seconds 0 put 3 get
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:keep0 0 tx 201326742 rx -- interval 10.0000 seconds 0 tx 134217878 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcache 0 hit 6 miss -- interval 10.0000 seconds 0 hit 3 miss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: fuseops 0 write 4 read -- interval 10.0000 seconds 0 write 2 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: blkio:0:0 0 write 32790 read -- interval 10.0000 seconds 0 write 16406 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Successfully read 1024 bytes from /keep/fe754b20898f553e034688333c3a4b7b+16068/14106407.CCXX.paired310.689141292.cram
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Successfully read 1024 bytes from /keep/fe754b20898f553e034688333c3a4b7b+16068/14106407.CCXX.paired310.689141292.cram.crai
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Preparing region list from chunk file [/keep/1bfc3b9a382ce0ab14c8e156745133c2+128+A55d05af2acd1fd610ea247d1015d61d10eda03e6@56b49aec/hs38DH.dict.32_of_400.region_list]
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Preparing fifos for output from 1 bcftools mpileup commands (one for each region) to bcftools concat
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Running ['cat', u'/tmp/crunch-job-task-work/humgen-01-02.19/14106407.CCXX.paired310.689141292.hs38DH.dict.32_of_400.region_list.part_1_of_1.g.vcf']
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Creating 'bcftools mpileup | bcftools norm' pipe for region 1/1 [chr2:5222314-13425498]
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Creating 'bcftools norm | tee' pipe for region 1/1 [chr2:5222314-13425498]
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Creating 'tee | bcftools view -h' pipe for region 1/1 [chr2:5222314-13425498]
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Creating 'tee | bcftools view' named pipe [/tmp/crunch-job-task-work/humgen-01-02.19/14106407.CCXX.paired310.689141292.hs38DH.dict.32_of_400.region_list.part_1_of_1.noheader.g.bcf] for region 1/1 [chr2:5222314-13425498]
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Opening concat fifo /tmp/crunch-job-task-work/humgen-01-02.19/14106407.CCXX.paired310.689141292.hs38DH.dict.32_of_400.region_list.part_1_of_1.g.vcf for writing
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Running ['bcftools-gvcf', 'mpileup', '-t', 'AD,INFO/AD', '-C50', '-pm2', '-F0.1', '-d10000', '--gvcf', '1,2,3,4,5,10,15', '-f', u'/keep/9a15a1d495a6efa0f3e05f1e851b694e+2227/hs38DH.fa', '-Ou', '-r', 'chr2:5222314-13425498', u'/keep/fe754b20898f553e034688333c3a4b7b+16068/14106407.CCXX.paired310.689141292.cram']
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Running ['bcftools', 'norm', '-f', u'/keep/9a15a1d495a6efa0f3e05f1e851b694e+2227/hs38DH.fa', '-Ou']
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Running ['teepot', u'/tmp/crunch-job-task-work/humgen-01-02.19/14106407.CCXX.paired310.689141292.hs38DH.dict.32_of_400.region_list.part_1_of_1.noheader.g.bcf', '-']
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Running ['bcftools', 'view', '-h', '-Oz', '-o', u'/tmp/crunch-job-task-work/humgen-01-02.19/14106407.CCXX.paired310.689141292.hs38DH.dict.32_of_400.region_list.part_1_of_1.headeronly.g.vcf.gz']
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr Running ['bcftools', 'view', '-H', '-Ov', u'/tmp/crunch-job-task-work/humgen-01-02.19/14106407.CCXX.paired310.689141292.hs38DH.dict.32_of_400.region_list.part_1_of_1.noheader.g.bcf']
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr bcftools mpileup 1/1 [chr2:5222314-13425498]: [mpileup] 1 samples in 1 input files
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: mem 220205056 cache 0 pgmajfault 281600000 rss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: cpu 2.8800 user 0.7800 sys 40 cpus -- interval 10.0002 seconds 2.6700 user 0.6200 sys
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:eth0 4618 tx 186639 rx -- interval 9.9999 seconds 0 tx 1852 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcalls 0 put 2840 get -- interval 10.0000 seconds 0 put 2834 get
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:keep0 0 tx 604973417 rx -- interval 10.0000 seconds 0 tx 403646675 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcache 2826 hit 14 miss -- interval 10.0000 seconds 2826 hit 8 miss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: fuseops 0 write 1688 read -- interval 10.0000 seconds 0 write 1684 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: blkio:0:0 0 write 219472701 read -- interval 10.0000 seconds 0 write 219439911 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: mem 220205056 cache 0 pgmajfault 281600000 rss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: cpu 2.9000 user 0.8100 sys 40 cpus -- interval 10.0004 seconds 0.0200 user 0.0300 sys
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:eth0 4618 tx 186639 rx -- interval 10.0005 seconds 0 tx 0 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcalls 0 put 2842 get -- interval 10.0000 seconds 0 put 2 get
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:keep0 0 tx 672082281 rx -- interval 10.0000 seconds 0 tx 67108864 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcache 2827 hit 15 miss -- interval 10.0000 seconds 1 hit 1 miss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: fuseops 0 write 1688 read -- interval 10.0000 seconds 0 write 0 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: blkio:0:0 0 write 219472701 read -- interval 10.0000 seconds 0 write 0 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr bcftools view -h 1/1 [chr2:5222314-13425498] completed successfully
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: mem 209993728 cache 0 pgmajfault 315219968 rss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: cpu 5.0900 user 0.9500 sys 40 cpus -- interval 9.9998 seconds 2.1900 user 0.1400 sys
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:eth0 4618 tx 186639 rx -- interval 9.9996 seconds 0 tx 0 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcalls 0 put 3805 get -- interval 10.0000 seconds 0 put 963 get
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:keep0 0 tx 873408873 rx -- interval 10.0000 seconds 0 tx 201326592 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcache 3788 hit 17 miss -- interval 10.0000 seconds 961 hit 2 miss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: fuseops 0 write 1918 read -- interval 10.0000 seconds 0 write 230 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: blkio:0:0 0 write 249881405 read -- interval 10.0000 seconds 0 write 30408704 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: mem 185712640 cache 0 pgmajfault 314929152 rss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: cpu 14.9900 user 1.2000 sys 40 cpus -- interval 10.0000 seconds 9.9000 user 0.2500 sys
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:eth0 4618 tx 186639 rx -- interval 10.0000 seconds 0 tx 0 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcalls 0 put 3829 get -- interval 10.0000 seconds 0 put 24 get
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:keep0 0 tx 940517737 rx -- interval 10.0000 seconds 0 tx 67108864 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcache 3811 hit 18 miss -- interval 10.0000 seconds 23 hit 1 miss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: fuseops 0 write 1935 read -- interval 10.0000 seconds 0 write 17 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: blkio:0:0 0 write 252109629 read -- interval 10.0000 seconds 0 write 2228224 read
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: mem 154304512 cache 0 pgmajfault 317513728 rss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: cpu 24.7200 user 1.5000 sys 40 cpus -- interval 10.0003 seconds 9.7300 user 0.3000 sys
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:eth0 4618 tx 186639 rx -- interval 10.0003 seconds 0 tx 0 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcalls 0 put 3850 get -- interval 10.0000 seconds 0 put 21 get
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:keep0 0 tx 1074735465 rx -- interval 10.0000 seconds 0 tx 134217728 rx
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcache 3830 hit 20 miss -- interval 10.0000 seconds 19 hit 2 miss
2016-01-22_13:22:26 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: fuseops 0 write 1952 read -- interval 10.0000 seconds 0 write 17 read
2016-01-22_13:22:27 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: blkio:0:0 0 write 253915965 read -- interval 10.0000 seconds 0 write 1806336 read
2016-01-22_13:22:27 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: mem 147116032 cache 0 pgmajfault 317861888 rss
2016-01-22_13:22:27 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: cpu 34.2600 user 1.7300 sys 40 cpus -- interval 10.0000 seconds 9.5400 user 0.2300 sys
2016-01-22_13:22:27 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:eth0 4774 tx 187097 rx -- interval 10.0002 seconds 156 tx 458 rx
2016-01-22_13:22:27 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcalls 0 put 3949 get -- interval 10.0000 seconds 0 put 99 get
2016-01-22_13:22:27 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: net:keep0 0 tx 1141844329 rx -- interval 10.0000 seconds 0 tx 67108864 rx
2016-01-22_13:22:27 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: keepcache 3929 hit 20 miss -- interval 10.0000 seconds 99 hit 0 miss
2016-01-22_13:22:27 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: fuseops 0 write 1979 read -- interval 10.0000 seconds 0 write 27 read
2016-01-22_13:22:27 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: blkio:0:0 0 write 256611133 read -- interval 10.0000 seconds 0 write 2695168 read
2016-01-22_13:22:27 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 stderr crunchstat: mem 150908928 cache 0 pgmajfault 318357504 rss

2016-01-22_13:22:28 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 killing orphaned srun process 31346 (task not in slurm queue, no stderr received)
2016-01-22_13:22:28 z8ta6-8i9sb-xb0qkz4qvfm5h8j 28675 19 sending 2x signal 2 to pid 31346