Bug #3052

[Crunch] stdout from non-task programs invoked by crunch-job should propagate to job log.

Added by Tom Clegg over 5 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Start date:
10/02/2014
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5

Description

See for example the log output for qr1hi-8i9sb-u6iqw0e7tgr9rc1


Subtasks

Task #4039: Review 3052-crunch-log-stdoutResolvedTim Pierce

Associated revisions

Revision 62790d76
Added by Peter Amstutz about 5 years ago

Merge branch '3052-crunch-log-stdout' closes #3052

History

#1 Updated by Tom Clegg over 5 years ago

  • Target version set to Arvados Future Sprints

#2 Updated by Ward Vandewege about 5 years ago

  • Description updated (diff)

#3 Updated by Ward Vandewege about 5 years ago

  • Subject changed from stdout from crunch tasks should propagate to job log. to [Crunch] stdout from crunch tasks should propagate to job log.

#4 Updated by Ward Vandewege about 5 years ago

  • Target version changed from Arvados Future Sprints to 2014-10-08 sprint

#5 Updated by Peter Amstutz about 5 years ago

  • Assigned To set to Peter Amstutz

#6 Updated by Peter Amstutz about 5 years ago

  • Status changed from New to In Progress

#7 Updated by Peter Amstutz about 5 years ago

  • The culprit is this code in #read_pipes of crunch-dispatch.rb:
          # Throw away child stdout
          begin
            j[:stdout].read_nonblock(2**20)
          rescue Errno::EAGAIN, EOFError
          end
    
  • Related to this, in crunch-job, tasks currently have their stdout and stderr fused into a single stream that goes to stderr. We want to restore the separation.

#8 Updated by Tom Clegg about 5 years ago

  • Subject changed from [Crunch] stdout from crunch tasks should propagate to job log. to [Crunch] stdout from non-task programs invoked by crunch-job should propagate to job log.

Peter Amstutz wrote:

  • Related to this, in crunch-job, tasks currently have their stdout and stderr fused into a single stream that goes to stderr. We want to restore the separation.

(but not here... added #4028)

#9 Updated by Tom Clegg about 5 years ago

(Recording question from IRC)

Why is this (a4679fb) better than combining the crunch-job child's stderr and stdout into a single unix pipe and leaving the existing buffer/propagation code alone?

This way, if a line arrives on stderr when stdout has printed a partial line, we get the intact stderr line followed by an intact stdout line, rather than a stdout line interrupted by a stderr line.

#10 Updated by Tim Pierce about 5 years ago

Review at a4679fb. Just one style nit:

crunch_dispatch.rb:337: can we use JSON dictionary style here? buf: {stdout: '', stderr: ''} If only because it always makes me dizzy when we combine dictionary styles in the same piece of code.

Otherwise LGTM!

#11 Updated by Anonymous about 5 years ago

  • Status changed from In Progress to Resolved

Applied in changeset arvados|commit:62790d76daf181a457d8f47741721967cd9cb7d9.

Also available in: Atom PDF