Arvados

• The culprit is this code in #read_pipes of crunch-dispatch.rb:
  

        # Throw away child stdout
        begin
          j[:stdout].read_nonblock(2**20)
        rescue Errno::EAGAIN, EOFError
        end
  

• Related to this, in crunch-job, tasks currently have their stdout and stderr fused into a single stream that goes to stderr. We want to restore the separation.

(Recording question from IRC)

Why is this (a4679fb) better than combining the crunch-job child's stderr and stdout into a single unix pipe and leaving the existing buffer/propagation code alone?

This way, if a line arrives on stderr when stdout has printed a partial line, we get the intact stderr line followed by an intact stdout line, rather than a stdout line interrupted by a stderr line.

Review at a4679fb. Just one style nit:

crunch_dispatch.rb:337: can we use JSON dictionary style here? buf: {stdout: '', stderr: ''} If only because it always makes me dizzy when we combine dictionary styles in the same piece of code.

Otherwise LGTM!