Arvados

Another instance with this error: qr1hi-d1hrv-96bdqhdlhgopth1

More errors: qr1hi-d1hrv-oykvsg3tkkvoznt

I created a tar archived that combines the requested Arvados SDK version and script version. That's 15144960 bytes. I ran it through just the tar extraction code of crunch-job:

#!/usr/bin/perl

if (!open(TARX, "|-", "tar", "-x")) {
  die "Error launching 'tar -xC $destdir': $!";
}
# If we send too much data to tar in one write (> 4-5 MiB), it stops, and we
# get SIGPIPE.  We must feed it data incrementally.
my $tar_input;
while (read(DATA, $tar_input, 65536)) {
  print TARX $tar_input;
}
if(!close(TARX)) {
  die "'tar -xC $destdir' exited $?: $!";
}

__DATA__

That worked. So right now I suspect an error passing the data over the wire via SLURM to the compute nodes.

This doesn't show the warning message "srun: error: compute32: task 0: Exited with exit code 141" but it does seem pretty close to what we're seeing:

9tee4:/tmp# n=10000000; ((set -o pipefail; echo 'exit 0;'; echo __DATA__; head -c$n /dev/urandom) | sudo -u crunch srun -n1 sh -c 'sleep 1; perl -'); echo $?
141

This command exits 0 with small n, but exits 141 when n is bigger than about 4000000.

Our install script exits without reading its __DATA__ section when this condition gets hit:

### Installation mode
open L, ">", "$destdir.lock" or die "$destdir.lock: $!";
flock L, LOCK_EX;
if (readlink ("$destdir.commit") eq $commit && -d $destdir) {
  # This version already installed -> nothing to do.
  exit(0);
}

6094-install-script-sigpipe @ b4c2a96 ensures the DATA section gets consumed even in that "don't need to extract" case, but I'm now less convinced this actually explains the failures.

Currently focusing on propagating the install script's stderr to the job log.

Reviewing e010ab73 (literally, just that commit)

It looks like you intend to use the recipe in select's perldoc to select specific filehandles for set periods of time. However, this commit does not assign 1 to vec's lvalue, which seems critical to making the recipe work. Is that an oversight? Or is this code intentionally doing something else? Right now it reads to me as:

my $bits = '';
# No-op: vec() returns a value, which we do nothing with.
# It has no other side effects.
vec($bits, fileno($install_stderr_r), 1);
# select('', undef, '', 0.1) - it's not clear what that will do, but I'm
# guessing it's a lot like select(undef, undef, undef, 0.1), a.k.a. sleep(0.1).
select ($bits, undef, $bits, 0.1);

If that's not right, well, I guess I'm just saying this code is awfully subtle.

Out of curiosity, did you test this anywhere? I'd be happy to kick the tires on my Docker cluster when I get to the office if that's helpful.

Thanks.

Brett Smith wrote:

this commit does not assign 1 to vec's lvalue

Whoops, yes, that was an oversight. Amended (now at c96f21f) by using the handy fhbits() sub that was sitting around unused.

Out of curiosity, did you test this anywhere? I'd be happy to kick the tires on my Docker cluster when I get to the office if that's helpful.

Tested on staging (http://curover.se/4xphq-8i9sb-zon7v8b0846mcwq), just by sticking "warn 'hola';" in the install script.

Looks like this:

2015-05-22_21:38:07 4xphq-8i9sb-zon7v8b0846mcwq 17571  Run install script on all workers
2015-05-22_21:38:07 4xphq-8i9sb-zon7v8b0846mcwq 17571  stderr starting: ['srun','--nodelist=compute1','-D','/tmp','--job-name=4xphq-8i9sb-zon7v8b0846mcwq','sh','-c','mkdir -p /tmp/crunch-job/opt && cd /tmp/crunch-job && perl -']
2015-05-22_21:38:07 4xphq-8i9sb-zon7v8b0846mcwq 17571  stderr hola at - line 129.
2015-05-22_21:38:07 4xphq-8i9sb-zon7v8b0846mcwq 17571  Install script exited 0

Tom Clegg wrote:

Brett Smith wrote:

this commit does not assign 1 to vec's lvalue

Whoops, yes, that was an oversight. Amended (now at c96f21f) by using the handy fhbits() sub that was sitting around unused.

c96f21fc is good to merge. Thanks.