Feature #8464

[Crunch2] Support stdout redirection

Added by Peter Amstutz about 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Radhika Chippada
Category:
-
Target version:
Start date:
05/09/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
0.5
Release:
Release relationship:
Auto

Description

https://dev.arvados.org/projects/arvados/wiki/Containers_API

"mounts": {
 "stdout":{
  "kind":"regular_file",
  "path":"/tmp/a.out" 
 }
}

Add a new field to ContainerRunner called StdoutFile of type io.WriteCloser.

  1. In SetupMounts, on first time through runner.ContainerRecord.Mounts, if bind is "stdout" then make sure it is of kind "file". Extend SetupMounts to support "file" kind.
  2. Before starting arv-mount, check that the "path" field under"stdout" starts with OutputPath. If not, it is an error
  3. strip off the leading "OutputPath" to get a relative path
  4. if there is a directory in the relative path, create the necessary subdirectories under "HostOutputDir"
  5. os.Open() the path for writing under HostOutputDir
  6. Assign the writer to ContainerRunner StdoutFile
  7. In AttachStreams, if StdoutFile is not nil, assign StdoutFile to runner.Stdout instead of NewThrottledLogger (will need to change type of ContainerRunner.Stdout to be an io.WriteCloser)

That should be it.


Subtasks

Task #9095: Review branch 8464-crunch2-stdoutResolvedPeter Amstutz

Associated revisions

Revision 44f0e83d
Added by Radhika Chippada almost 5 years ago

closes #8464
Merge branch '8464-crunch2-stdout'

History

#1 Updated by Tom Clegg about 5 years ago

  • Target version set to 2016-05-11 sprint

#2 Updated by Tom Clegg about 5 years ago

  • Assigned To set to Peter Amstutz

#3 Updated by Tom Clegg about 5 years ago

  • Story points set to 0.5

#4 Updated by Radhika Chippada about 5 years ago

  • Assigned To changed from Peter Amstutz to Radhika Chippada

#5 Updated by Peter Amstutz about 5 years ago

  • Description updated (diff)

#6 Updated by Radhika Chippada about 5 years ago

  • Status changed from New to In Progress

#7 Updated by Radhika Chippada about 5 years ago

  • Description updated (diff)

#8 Updated by Peter Amstutz almost 5 years ago

I don't know what this block starting at L316 is doing, and it won't work because (a) the OutputPath inside the container is different from the HostOutputDir and (b) the HostOutputDir may not exist yet if it hasn't processed all the mounts yet. I think you can just remove the code entirely.

} else if mnt.Kind == "file" {
...
}

In AttachStreams(), runner.ContainerRecord.Mounts is a map, so you don't need to iterate over it. You can just say

if mnt, ok := runner.ContainerRecord.Mounts["stdout"]; ok {
...
}

L429: Should be path.Join(runner.HostOutputDir, stdoutSubdirs)

L434: It's not necessary to call os.Stat() after os.MkdirAll() if you're going to ignore the return value.

L437: also should be path.Join(runner.HostOutputDir, stdoutPath)

#9 Updated by Radhika Chippada almost 5 years ago

73550248

I don't know what this block starting at L316 is doing

Removed this and updated the tests

In AttachStreams(), runner.ContainerRecord.Mounts is a map, so you don't need to iterate over it. You can just say

Right, updated it

L429: Should be path.Join(runner.HostOutputDir, stdoutSubdirs)
L437: also should be path.Join(runner.HostOutputDir, stdoutPath)

Updated

L434: It's not necessary to call os.Stat() after os.MkdirAll() if you're going to ignore the return value.

Yes, I had this during my test / debug cycle and failed to see it afterwards. Removed it.

Thanks.

#10 Updated by Radhika Chippada almost 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100

Applied in changeset arvados|commit:44f0e83d50f688bf73c336747402d490346f5c34.

Also available in: Atom PDF