Arvados

There are other pipeline instances where it complains with that error in the logs, e.g. qr1hi-d1hrv-wji6s57f4k3sqix. Weirdly, the pipeline instance status is 'failed' though the 2 jobs in the pipeline are listed as 'success'.

I could be wrong, but it appears that "someone" is using the pipeline instance as pipeline template? If you look at the log for pipeline_instances/qr1hi-d1hrv-d6ix17kcd2fuu0d, it says "2014-10-07 18:41:41 +0000 -- pipeline_template qr1hi-d1hrv-d6ix17kcd2fuu0d Errors . . . " where the uuid printed is the instance uuid, not the template uuid.

I pointed my dev env to qr1hi and inserted some debug statements and the value for "b" is definitely there at least during the creation of the instance (I used --submit as well as --run-pipeline-here) and in both cases the value of "b" is the one provided as "value" in the pipeline template.

However, it appears that it is not found when it is actually run (later when it is scheduled in case of --submit for example) and the uuid shown in log tab looks suspicious (instance's instead of template's).

Some more observations about this issue.

- The pipeline instance being created from the template seems correct --- :b=>{:value=>{:group=>"e", :regex=>"(.*)_[12]\\.fastq\\.gz$"}}

#<PipelineInstance:0x00000004f2c160 @attributes_to_update={}, @pi={:href=>"/pipeline_instances/qr1hi-d1hrv-jsx0xg30okarj68", :kind=>"arvados#pipelineInstance", :etag=>"115l0h2qo261d4djsjgpivoti", :uuid=>"qr1hi-d1hrv-jsx0xg30okarj68", :owner_uuid=>"qr1hi-tpzed-ktpvhqu89qoib9f", :created_at=>"2014-10-08T02:11:06Z", :modified_by_client_uuid=>"qr1hi-ozdt8-obw7foaks3qjyej", :modified_by_user_uuid=>"qr1hi-tpzed-ktpvhqu89qoib9f", :modified_at=>"2014-10-08T02:11:07Z", :pipeline_template_uuid=>"qr1hi-p5p6p-ujw7clygo3vtorv", :pipeline_template=>{:href=>"/pipeline_templates/qr1hi-p5p6p-ujw7clygo3vtorv", :kind=>"arvados#pipelineTemplate", :etag=>"bf90157j7q69dlqqwfczwzuvn", :uuid=>"qr1hi-p5p6p-ujw7clygo3vtorv", :owner_uuid=>"qr1hi-tpzed-rv6f0l8lbvdzkog", :created_at=>"2014-09-10T17:49:08Z", :modified_by_client_uuid=>"qr1hi-ozdt8-obw7foaks3qjyej", :modified_by_user_uuid=>"qr1hi-tpzed-rv6f0l8lbvdzkog", :modified_at=>"2014-10-07T18:41:10Z", :name=>"mxn test", :components=>{:first=>{:script=>"run-command", :script_version=>"4042-run-command-MxN", :repository=>"bcosc", :script_parameters=>{:command=>["bwa", "mem", "-t", "$(node.cores)", "-R", "@RG\\tID:$(basename $(eachread))\\tPL:illumina\\tSM:$(sample_id)", "$(glob $(dir $(reference))/*.fasta)", {:foreach=>"b", :command=>["$(glob $(b)/*_1.fastq*)", "$(glob $(b)/*_2.fastq*)"]}], :reference=>{:required=>true, :dataclass=>"Collection", :default=>"3514b8e5da0e8d109946bc809b20a78a+5698", :link_name=>"human_g1k_v37 reference data", :title=>"Reference genome (FASTA)"}, :sample_id=>{:required=>true, :default=>"my_sample", :dataclass=>"text", :title=>"Sample identifier"}, :reads=>{:output_of=>"split"}, :"task.foreach"=>["b"], :"task.stdout"=>"$(basename $(basename $(glob $(eachread)/*_1.fastq*))).sam", :b=>{:value=>{:group=>"e", :regex=>"(.*)_[12]\\.fastq\\.gz$"}}, :d=>{:required=>true, :default=>"d0136bc494c21f79fc1b6a390561e6cb+2778", :title=>"input collection", :dataclass=>"Collection"}, :e=>"$(dir $(d))"}, :runtime_constraints=>{:docker_image=>"arvados/jobs-java-bwa-samtools"}, :output_name=>false}}, :description=>nil}, :name=>nil, :components=>{:first=>{:script=>"run-command", :script_version=>"4042-run-command-MxN", :repository=>"bcosc", :script_parameters=>{:command=>["bwa", "mem", "-t", "$(node.cores)", "-R", "@RG\\tID:$(basename $(eachread))\\tPL:illumina\\tSM:$(sample_id)", "$(glob $(dir $(reference))/*.fasta)", {:foreach=>"b", :command=>["$(glob $(b)/*_1.fastq*)", "$(glob $(b)/*_2.fastq*)"]}], :reference=>"3514b8e5da0e8d109946bc809b20a78a+5698", :sample_id=>"my_sample", :reads=>{:output_of=>"split"}, :"task.foreach"=>["b"], :"task.stdout"=>"$(basename $(basename $(glob $(eachread)/*_1.fastq*))).sam", :b=>nil, :d=>"d0136bc494c21f79fc1b6a390561e6cb+2778", :e=>"$(dir $(d))"}, :runtime_constraints=>{:docker_image=>"arvados/jobs-java-bwa-samtools"}, :output_name=>false}}, :dependencies=>["3514b8e5da0e8d109946bc809b20a78a+5698", "d0136bc494c21f79fc1b6a390561e6cb+2778"], :properties=>{:run_options=>{:enable_job_reuse=>true}}, :state=>"RunningOnClient", :components_summary=>{:todo=>1, :done=>0, :failed=>0}, :description=>"Created at 2014-10-07 22:11:06 -0400 using the pipeline template *mxn test*", :started_at=>"2014-10-08T02:11:07Z", :finished_at=>nil}>

- However, when the instance is fetched to run, the apply_parameters method in a-r-p-i is not able to see the value of "b".

- We have the following starting at line 437

        parameter = { :value => parameter } unless parameter.is_a? Hash
        value =
          (params["#{componentname}::#{parametername}"] ||
           parameter[:value] ||
           (parameter[:output_of].nil? &&
            (params[parametername.to_s] ||
             parameter[:default])) ||
           nil)

- I think we need the following (between lines 444 and 445) to account for the case where the "value" is a Hash:

value = component if value.nil? && !component.nil?

While on the subject, I think we should fix the log message as well. Currently, the log message looks as:

2014-10-07 18:41:41 +0000 -- pipeline_template qr1hi-d1hrv-d6ix17kcd2fuu0d
Errors:
first::b - required parameter is missing

- We should fix it to say

 2014-10-07 18:41:41 +0000 -- pipeline qr1hi-d1hrv-d6ix17kcd2fuu0d 

I think this is (partly) a pipeline template bug.

There are two ways to specify a parameter for a pipeline component.

1. "paramname":"xyzzy" -- this is a shortcut only suitable for undocumented non-user-editable parameters.
2. "paramname":{"value":"xyzzy","required":true,"title":"Magic spell to use"} -- this is the general form.

The first form cannot be used when the parameter value is a hash, because it cannot be distinguished from the second form.

Radhika Chippada wrote:

While on the subject, I think we should fix the log message as well. Currently, the log message looks as:

[...]

- We should fix it to say [...] or use the word template or instance correctly depending on the object class.

Yes, there are several instances in the code where we say "template" even though @template is sometimes just a copy of @instance. We should detect this, and say "template" or "instance" depending on which it is.

I was able to create an instance from the template and execute a run-jobs-here command without getting "first::b - required parameter is missing" error by updating the template as listed below.

Please review these changes. Here is what changed:

- Tom updated arv-run-pipeline-instace:

        component[:script_parameters][parametername] =
          parameter.dup.merge(value: value)

- I updated the template as follows:

     "b":{
      "value":{
       "group":"e",
       "regex":"(.*)_[12]\\.fastq\\.gz$" 
      }