https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422014-10-07T19:54:57ZArvadosArvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=160692014-10-07T19:54:57ZWard Vandewegeward@curii.com
<ul></ul><p>There are other pipeline instances where it complains with that error in the logs, e.g. <a href="https://arvadosapi.com/qr1hi-d1hrv-wji6s57f4k3sqix">qr1hi-d1hrv-wji6s57f4k3sqix</a>. Weirdly, the pipeline instance status is 'failed' though the 2 jobs in the pipeline are listed as 'success'.</p> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=160712014-10-07T19:58:59ZWard Vandewegeward@curii.com
<ul><li><strong>Target version</strong> set to <i>Bug Triage</i></li></ul> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=160992014-10-07T20:09:16ZRadhika Chippadaradhika@curoverse.com
<ul><li><strong>Subject</strong> changed from <i>Parameter in pipeline runner is missing</i> to <i>[SDKs] Parameter in pipeline runner is missing</i></li><li><strong>Category</strong> set to <i>SDKs</i></li></ul><p>I think it is arv-run-pipeline-instance issue and so I set the category as SDKs.</p> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=161262014-10-07T21:59:10ZRadhika Chippadaradhika@curoverse.com
<ul></ul><p>I could be wrong, but it appears that "someone" is using the pipeline instance as pipeline template? If you look at the log for pipeline_instances/qr1hi-d1hrv-d6ix17kcd2fuu0d, it says "2014-10-07 18:41:41 +0000 -- pipeline_template <a href="https://arvadosapi.com/qr1hi-d1hrv-d6ix17kcd2fuu0d">qr1hi-d1hrv-d6ix17kcd2fuu0d</a> Errors . . . " where the uuid printed is the instance uuid, not the template uuid.</p>
<p>I pointed my dev env to qr1hi and inserted some debug statements and the value for "b" is definitely there at least during the creation of the instance (I used --submit as well as --run-pipeline-here) and in both cases the value of "b" is the one provided as "value" in the pipeline template.</p>
<p>However, it appears that it is not found when it is actually run (later when it is scheduled in case of --submit for example) and the uuid shown in log tab looks suspicious (instance's instead of template's).</p> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=161272014-10-08T12:59:57ZRadhika Chippadaradhika@curoverse.com
<ul></ul><p>Some more observations about this issue.</p>
<p>- The pipeline instance being created from the template seems correct --- <code>:b=>{:value=>{:group=>"e", :regex=>"(.*)_[12]\\.fastq\\.gz$"}}</code></p>
<pre>
#<PipelineInstance:0x00000004f2c160 @attributes_to_update={}, @pi={:href=>"/pipeline_instances/qr1hi-d1hrv-jsx0xg30okarj68", :kind=>"arvados#pipelineInstance", :etag=>"115l0h2qo261d4djsjgpivoti", :uuid=>"<a href="https://arvadosapi.com/qr1hi-d1hrv-jsx0xg30okarj68">qr1hi-d1hrv-jsx0xg30okarj68</a>", :owner_uuid=>"<a href="https://arvadosapi.com/qr1hi-tpzed-ktpvhqu89qoib9f">qr1hi-tpzed-ktpvhqu89qoib9f</a>", :created_at=>"2014-10-08T02:11:06Z", :modified_by_client_uuid=>"<a href="https://arvadosapi.com/qr1hi-ozdt8-obw7foaks3qjyej">qr1hi-ozdt8-obw7foaks3qjyej</a>", :modified_by_user_uuid=>"<a href="https://arvadosapi.com/qr1hi-tpzed-ktpvhqu89qoib9f">qr1hi-tpzed-ktpvhqu89qoib9f</a>", :modified_at=>"2014-10-08T02:11:07Z", :pipeline_template_uuid=>"<a href="https://arvadosapi.com/qr1hi-p5p6p-ujw7clygo3vtorv">qr1hi-p5p6p-ujw7clygo3vtorv</a>", :pipeline_template=>{:href=>"/pipeline_templates/qr1hi-p5p6p-ujw7clygo3vtorv", :kind=>"arvados#pipelineTemplate", :etag=>"bf90157j7q69dlqqwfczwzuvn", :uuid=>"<a href="https://arvadosapi.com/qr1hi-p5p6p-ujw7clygo3vtorv">qr1hi-p5p6p-ujw7clygo3vtorv</a>", :owner_uuid=>"<a href="https://arvadosapi.com/qr1hi-tpzed-rv6f0l8lbvdzkog">qr1hi-tpzed-rv6f0l8lbvdzkog</a>", :created_at=>"2014-09-10T17:49:08Z", :modified_by_client_uuid=>"<a href="https://arvadosapi.com/qr1hi-ozdt8-obw7foaks3qjyej">qr1hi-ozdt8-obw7foaks3qjyej</a>", :modified_by_user_uuid=>"<a href="https://arvadosapi.com/qr1hi-tpzed-rv6f0l8lbvdzkog">qr1hi-tpzed-rv6f0l8lbvdzkog</a>", :modified_at=>"2014-10-07T18:41:10Z", :name=>"mxn test", :components=>{:first=>{:script=>"run-command", :script_version=>"4042-run-command-MxN", :repository=>"bcosc", :script_parameters=>{:command=>["bwa", "mem", "-t", "$(node.cores)", "-R", "@RG\\tID:$(basename $(eachread))\\tPL:illumina\\tSM:$(sample_id)", "$(glob $(dir $(reference))/*.fasta)", {:foreach=>"b", :command=>["$(glob $(b)/*_1.fastq*)", "$(glob $(b)/*_2.fastq*)"]}], :reference=>{:required=>true, :dataclass=>"Collection", :default=>"3514b8e5da0e8d109946bc809b20a78a+5698", :link_name=>"human_g1k_v37 reference data", :title=>"Reference genome (FASTA)"}, :sample_id=>{:required=>true, :default=>"my_sample", :dataclass=>"text", :title=>"Sample identifier"}, :reads=>{:output_of=>"split"}, :"task.foreach"=>["b"], :"task.stdout"=>"$(basename $(basename $(glob $(eachread)/*_1.fastq*))).sam", :b=>{:value=>{:group=>"e", :regex=>"(.*)_[12]\\.fastq\\.gz$"}}, :d=>{:required=>true, :default=>"d0136bc494c21f79fc1b6a390561e6cb+2778", :title=>"input collection", :dataclass=>"Collection"}, :e=>"$(dir $(d))"}, :runtime_constraints=>{:docker_image=>"arvados/jobs-java-bwa-samtools"}, :output_name=>false}}, :description=>nil}, :name=>nil, :components=>{:first=>{:script=>"run-command", :script_version=>"4042-run-command-MxN", :repository=>"bcosc", :script_parameters=>{:command=>["bwa", "mem", "-t", "$(node.cores)", "-R", "@RG\\tID:$(basename $(eachread))\\tPL:illumina\\tSM:$(sample_id)", "$(glob $(dir $(reference))/*.fasta)", {:foreach=>"b", :command=>["$(glob $(b)/*_1.fastq*)", "$(glob $(b)/*_2.fastq*)"]}], :reference=>"3514b8e5da0e8d109946bc809b20a78a+5698", :sample_id=>"my_sample", :reads=>{:output_of=>"split"}, :"task.foreach"=>["b"], :"task.stdout"=>"$(basename $(basename $(glob $(eachread)/*_1.fastq*))).sam", :b=>nil, :d=>"d0136bc494c21f79fc1b6a390561e6cb+2778", :e=>"$(dir $(d))"}, :runtime_constraints=>{:docker_image=>"arvados/jobs-java-bwa-samtools"}, :output_name=>false}}, :dependencies=>["3514b8e5da0e8d109946bc809b20a78a+5698", "d0136bc494c21f79fc1b6a390561e6cb+2778"], :properties=>{:run_options=>{:enable_job_reuse=>true}}, :state=>"RunningOnClient", :components_summary=>{:todo=>1, :done=>0, :failed=>0}, :description=>"Created at 2014-10-07 22:11:06 -0400 using the pipeline template *mxn test*", :started_at=>"2014-10-08T02:11:07Z", :finished_at=>nil}>
</pre>
<p>- However, when the instance is fetched to run, the apply_parameters method in a-r-p-i is not able to see the value of "b".</p>
<p>- We have the following starting at line 437<br /><pre>
parameter = { :value => parameter } unless parameter.is_a? Hash
value =
(params["#{componentname}::#{parametername}"] ||
parameter[:value] ||
(parameter[:output_of].nil? &&
(params[parametername.to_s] ||
parameter[:default])) ||
nil)
</pre></p>
<p>- I think we need the following (between lines 444 and 445) to account for the case where the "value" is a Hash:<br /><pre>
value = component if value.nil? && !component.nil?
</pre></p> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=161282014-10-08T13:05:12ZRadhika Chippadaradhika@curoverse.com
<ul></ul><p>While on the subject, I think we should fix the log message as well. Currently, the log message looks as:</p>
<pre>
2014-10-07 18:41:41 +0000 -- pipeline_template <a href="https://arvadosapi.com/qr1hi-d1hrv-d6ix17kcd2fuu0d">qr1hi-d1hrv-d6ix17kcd2fuu0d</a>
Errors:
first::b - required parameter is missing
</pre>
<p>- We should fix it to say <pre> 2014-10-07 18:41:41 +0000 -- pipeline <a href="https://arvadosapi.com/qr1hi-d1hrv-d6ix17kcd2fuu0d">qr1hi-d1hrv-d6ix17kcd2fuu0d</a> </pre> or use the word template or instance correctly depending on the object class.</p> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=161342014-10-08T13:36:16ZTom Cleggtom@curii.com
<ul></ul><p>I think this is (partly) a pipeline template bug.</p>
<p>There are two ways to specify a parameter for a pipeline component.</p>
<ol>
<li><code>"paramname":"xyzzy"</code> -- this is a shortcut only suitable for undocumented non-user-editable parameters.</li>
<li><code>"paramname":{"value":"xyzzy","required":true,"title":"Magic spell to use"}</code> -- this is the general form.</li>
</ol>
<p>The first form cannot be used when the parameter value is a hash, because it cannot be distinguished from the second form.</p>
<p>Radhika Chippada wrote:</p>
<blockquote>
<p>While on the subject, I think we should fix the log message as well. Currently, the log message looks as:</p>
<p>[...]</p>
<p>- We should fix it to say [...] or use the word template or instance correctly depending on the object class.</p>
</blockquote>
<p>Yes, there are several instances in the code where we say "template" even though @template is sometimes just a copy of @instance. We should detect this, and say "template" or "instance" depending on which it is.</p> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=161372014-10-08T13:55:41ZRadhika Chippadaradhika@curoverse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>Assigned To</strong> set to <i>Radhika Chippada</i></li><li><strong>Target version</strong> changed from <i>Bug Triage</i> to <i>2014-10-08 sprint</i></li></ul> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=161602014-10-08T17:28:59ZWard Vandewegeward@curii.com
<ul><li><strong>Story points</strong> set to <i>0.5</i></li></ul> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=162462014-10-08T19:21:15ZWard Vandewegeward@curii.com
<ul><li><strong>Target version</strong> changed from <i>2014-10-08 sprint</i> to <i>2014-10-29 sprint</i></li></ul> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=163512014-10-09T20:47:55ZRadhika Chippadaradhika@curoverse.com
<ul></ul><p>I was able to create an instance from the template and execute a run-jobs-here command without getting "first::b - required parameter is missing" error by updating the template as listed below.</p>
<p>Please review these changes. Here is what changed:</p>
<p>- Tom updated arv-run-pipeline-instace:<br /><pre>
component[:script_parameters][parametername] =
parameter.dup.merge(value: value)
</pre></p>
<p>- I updated the template as follows:<br /><pre>
"b":{
"value":{
"group":"e",
"regex":"(.*)_[12]\\.fastq\\.gz$"
}
</pre></p> Arvados - Bug #4126: [SDKs] Parameter in pipeline runner is missinghttps://dev.arvados.org/issues/4126?journal_id=163682014-10-10T13:50:08ZRadhika Chippadaradhika@curoverse.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset arvados|commit:ff9d78bbeb09ca917f0814c6055c47db5a4e7e4a.</p>