Arvados

You can get the same functionality today by running arv pipeline_template create --pipeline_template "$(cat input_file.json)".

Brett, it's true that the workaround works for small payloads, but for larger ones you wind up with:

arv: Argument list too long

(trying to use `arv` to create a collection with a large manifest that can't be fit onto the command line)

I have implemented a simple modification to the CLI that allows supplying of the request body from a file (or stdin) rather than requiring it be specified on the command line. This is implemented by first attempting to use `JSON.parse` to parse the supplied value request body argument as JSON. If JSON parsing succeeds, the value is handled as before and sent as the request body. If parsing fails, we assume the string is actually a filename (or the special value '-' to indicate stdin) and read the contents of that file (or stdin) and use that as the request body instead.

https://github.com/curoverse/arvados/pull/43

With the changes in the PR, all of the following would be equivalent:

arv pipeline_template create --pipeline_template "$(cat input_file.json)" 
arv pipeline_template create --pipeline_template input_file.json
cat input_file.json | arv pipeline_template create --pipeline_template -

But the latter two will work for arbitrarily large `input_file.json`, whereas the first one will be limited to the maximum command line length.

Updated PR 43 (https://github.com/curoverse/arvados/pull/43) based on discussion on IRC. Adds additional safety check - refuses to proceed if the provided argument is both valid JSON and also a readable file.

Review comments for branch wtsi-hgi-8087-arv-cli-request-body-from-file

• The error message displayed when a wrong json is provided at command line is misleading. It should say “invalid json”. May be we can say neither valid json nor existing filename.
  

  arv pipeline_template create --pipeline-template '{"name":"does not end properly}'
  File '{"name":"does not end properly}' specified for option '--pipeline_template' does not exist.
  

• When the “yes it is readable file” path is being executed, can we please check if the resource_body from the file is a valid json or not? There is no point in sending the request to the API server when the file content is not a valid json. ( else block at line # 660 )

• This is not really an issue with functionality. But can you please rename _is_json, _resource_opt_desc etc to not start with an underscore. As far as I know, we do not use this naming convention. I do appreciate the descriptive names though :)

• Can we not print the potentially long string twice here (suggest removing the first #{resource_body} to make reading the message easier.
  • Current: Argument '#{resource_body}' specified for option '--#{resource_schema.to_sym}' is both valid JSON and a readable file, cannot continue (suggest renaming the file '#{resource_body}
  • Suggested: Argument specified for option '--#{resource_schema.to_sym}' is both valid JSON and a readable file. Aborting. Please consider renaming the file '#{resource_body}

• Trailing white space for this comment line “we don't actually need the results of the parsing”

• Just an observation. The example command listed in note 7 should use --pipeline-template ( not _ )

• Would it be too much if I make the observation that we do not have an automated test for this? Please consider adding a test in test_arv-collection-create.rb. If your time does not permit, I can try to add it little later in the sprint.

Thanks.