Feature #8464
closed
[Crunch2] Support stdout redirection
Description
https://dev.arvados.org/projects/arvados/wiki/Containers_API
"mounts": {
"stdout":{
"kind":"regular_file",
"path":"/tmp/a.out"
}
}
Add a new field to ContainerRunner called StdoutFile of type io.WriteCloser.
- In
SetupMounts, on first time through runner.ContainerRecord.Mounts, ifbindis "stdout" then make sure it is of kind "file". Extend SetupMounts to support "file" kind. - Before starting arv-mount, check that the "path" field under"stdout" starts with OutputPath. If not, it is an error
- strip off the leading "OutputPath" to get a relative path
- if there is a directory in the relative path, create the necessary subdirectories under "HostOutputDir"
- os.Open() the path for writing under HostOutputDir
- Assign the writer to ContainerRunner StdoutFile
- In
AttachStreams, if StdoutFile is not nil, assign StdoutFile torunner.Stdoutinstead of NewThrottledLogger (will need to change type of ContainerRunner.Stdout to be an io.WriteCloser)
That should be it.
Updated by
Updated by Radhika Chippada over 8 years ago
- Assigned To changed from Peter Amstutz to Radhika Chippada
Updated by Radhika Chippada over 8 years ago
- Status changed from New to In Progress
Updated by Peter Amstutz over 8 years ago
I don't know what this block starting at L316 is doing, and it won't work because (a) the OutputPath inside the container is different from the HostOutputDir and (b) the HostOutputDir may not exist yet if it hasn't processed all the mounts yet. I think you can just remove the code entirely.
} else if mnt.Kind == "file" {
...
}
In AttachStreams(), runner.ContainerRecord.Mounts is a map, so you don't need to iterate over it. You can just say
if mnt, ok := runner.ContainerRecord.Mounts["stdout"]; ok {
...
}
L429: Should be path.Join(runner.HostOutputDir, stdoutSubdirs)
L434: It's not necessary to call os.Stat() after os.MkdirAll() if you're going to ignore the return value.
L437: also should be path.Join(runner.HostOutputDir, stdoutPath)
Updated by Radhika Chippada over 8 years ago
I don't know what this block starting at L316 is doing
Removed this and updated the tests
In AttachStreams(), runner.ContainerRecord.Mounts is a map, so you don't need to iterate over it. You can just say
Right, updated it
L429: Should be path.Join(runner.HostOutputDir, stdoutSubdirs)
L437: also should be path.Join(runner.HostOutputDir, stdoutPath)
Updated
L434: It's not necessary to call os.Stat() after os.MkdirAll() if you're going to ignore the return value.
Yes, I had this during my test / debug cycle and failed to see it afterwards. Removed it.
Thanks.
Updated by Radhika Chippada over 8 years ago
- Status changed from In Progress to Resolved
- % Done changed from 0 to 100
Applied in changeset arvados|commit:44f0e83d50f688bf73c336747402d490346f5c34.