Arvados: Issueshttps://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422023-01-20T16:25:15ZArvados
Redmine Arvados - Bug #19967 (Resolved): crunch-run periodically updates container costhttps://dev.arvados.org/issues/199672023-01-20T16:25:15ZTom Cleggtom@curii.com
<p>Currently the container cost is recorded as 0 in this case.</p>
<p>This should ensure the cost of a cancel-retry-cancel-retry-cancel-fail cycle is reflected properly.</p>
<p>Have crunch-run periodically update the price while the container is running, perhaps each time it gets a SIGUSR2 after loadPrices (i.e., on each dispatcher probe).</p> Tapestry - Feature #8701 (Closed): Arvados job/pipeline generates ClinVar HTML report from VCFhttps://dev.arvados.org/issues/87012016-03-15T17:59:55ZTom Cleggtom@curii.comTapestry - Feature #8700 (Closed): Arvados job/pipeline generates GET-Evidence HTML report from VCFhttps://dev.arvados.org/issues/87002016-03-15T17:59:43ZTom Cleggtom@curii.comArvados - Task #6116 (Closed): Review 5883-oauth2-update at get-evidence.githttps://dev.arvados.org/issues/61162015-05-20T20:12:49ZTom Cleggtom@curii.comArvados - Bug #3720 (Resolved): [API] Fetching collection record is very slow when manifest is no...https://dev.arvados.org/issues/37202014-08-27T14:59:46ZTom Cleggtom@curii.com
<p>This makes nginx give up on Workbench and say "502 Bad Gateway".</p> Arvados - Feature #3585 (Resolved): [Crunch] arv-run-pipeline-instance --template should support ...https://dev.arvados.org/issues/35852014-08-13T13:41:43ZTom Cleggtom@curii.comArvados - Task #3374 (Resolved): Test patched Python packagehttps://dev.arvados.org/issues/33742014-07-26T15:27:51ZTom Cleggtom@curii.comArvados - Idea #3078 (Resolved): arv-put adds a name link after creating a collectionhttps://dev.arvados.org/issues/30782014-06-23T10:37:30ZTom Cleggtom@curii.com
arv-put needs two new command line arguments.
<ol>
<li><code>--name</code> → an arbitrary string</li>
<li><code>--project-uuid</code> → uuid of a project (group)</li>
</ol>
New behavior:
<ul>
<li>If neither argument is provided, arv-put should display a warning like: "Warning: with no --name or --project-uuid specified, files will be stored in Keep as cached data and will be harder for you to find later." </li>
<li>If a name is provided, but not a project uuid, there is not enough information to create a name link so arv-put should fail immediately with an explanation.</li>
<li>If a project uuid is provided, but not a name, the name should default to "Collection saved by {user}@{hostname} at {current date/time}".</li>
<li>If both arguments are provided, after creating the collection object, arv-put should create a name link.
<ul>
<li>owner_uuid = tail_uuid = the specified project uuid</li>
<li>head_uuid = the new collection uuid</li>
<li>link_class = "name" </li>
<li>name = the specified name</li>
</ul>
</li>
<li>If everything else succeeds but link creation fails, output the collection's hash on stdout as usual, but display an error message and exit non-zero.</li>
</ul>
Do not (yet):
<ul>
<li>Accept a project <em>name</em> on the command line</li>
<li>Guess a project if none is given</li>
</ul> Arvados - Idea #2880 (Resolved): Component/job can specify minimum memory and scratch space for w...https://dev.arvados.org/issues/28802014-05-27T14:23:13ZTom Cleggtom@curii.com
<a name="Implement-runtime-constraints"></a>
<h3 >Implement runtime constraints<a href="#Implement-runtime-constraints" class="wiki-anchor">¶</a></h3>
<p>The runtime constraints are currently specced as:</p>
<ul>
<li><code>min_ram_mb_per_node</code> - The amount of RAM (MiB) available on each Node.</li>
<li><code>min_scratch_mb_per_node</code> - The amount of disk space (MiB) available for local caching on each Node.</li>
</ul>
<p>For now, crunch-dispatch should start the biggest job that it can given the Nodes that are currently available.</p> Arvados - Idea #2879 (Resolved): Crunch uses Docker images stored in Keep, and records Keep locat...https://dev.arvados.org/issues/28792014-05-27T14:18:23ZTom Cleggtom@curii.com
<p>Description:</p>
<ul>
<li>Users put Docker images in Keep with a one-liner, hopefully <code>docker save imagename | arv-put</code>.</li>
<li>Users specify that Collection locator as the <code>docker_image</code> constraint on a Job.</li>
<li>crunch-job installs images from Keep rather than using <code>docker pull</code>.</li>
<li>crunch-job records the Keep Collection locator used in a new Jobs column (to enable us to support symbolic names in the runtime constraints later).</li>
<li>Crunch uses information from the new column to decide whether or not a Job is reusable.</li>
</ul>
<p>Reach goals:</p>
<ul>
<li>Support Docker tags or similar functionality in the <code>docker_image</code> runtime constraint. (This gets translated so that only Keep Collection locators get stored in the new column.)</li>
<li>Provide a two-way mapping between Docker image hashes and Collection locators.</li>
</ul> Arvados - Bug #2800 (Resolved): [Crunch] Remove global state in Python SDKhttps://dev.arvados.org/issues/28002014-05-09T03:03:04ZTom Cleggtom@curii.com
<p>It should be easy for a Python program to communicate with API and Keep storage servers at multiple sites in one process. For example:</p>
<pre><code class="python syntaxhl"><span class="n">a</span> <span class="o">=</span> <span class="n">arvados</span><span class="p">.</span><span class="n">api</span><span class="p">(</span><span class="s">'v1'</span><span class="p">,</span> <span class="n">host</span><span class="o">=</span><span class="s">'localhost'</span><span class="p">,</span> <span class="n">token</span><span class="o">=</span><span class="s">'12345'</span><span class="p">,</span> <span class="n">insecure</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span>
<span class="n">b</span> <span class="o">=</span> <span class="n">arvados</span><span class="p">.</span><span class="n">api</span><span class="p">(</span><span class="s">'v1'</span><span class="p">,</span> <span class="n">host</span><span class="o">=</span><span class="s">'qr1hi.arvadosapi.com'</span><span class="p">,</span> <span class="n">token</span><span class="o">=</span><span class="s">'xyzzy'</span><span class="p">)</span>
<span class="n">x</span> <span class="o">=</span> <span class="n">a</span><span class="p">.</span><span class="n">collections</span><span class="p">().</span><span class="nb">list</span><span class="p">().</span><span class="n">execute</span><span class="p">()</span>
<span class="n">w1</span> <span class="o">=</span> <span class="n">CollectionWriter</span><span class="p">(</span><span class="n">client</span><span class="o">=</span><span class="n">a</span><span class="p">)</span>
<span class="n">w2</span> <span class="o">=</span> <span class="n">CollectionWriter</span><span class="p">(</span><span class="n">client</span><span class="o">=</span><span class="n">b</span><span class="p">)</span>
<span class="n">w1</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="s">'foo'</span><span class="p">)</span>
<span class="n">w2</span><span class="p">.</span><span class="n">write</span><span class="p">(</span><span class="s">'bar'</span><span class="p">)</span>
</code></pre> Arvados - Bug #2759 (Resolved): Cannot view collection in workbenchhttps://dev.arvados.org/issues/27592014-05-07T13:17:59ZTom Cleggtom@curii.com
<p><a class="external" href="https://workbench.4xphq.arvadosapi.com/jobs/4xphq-8i9sb-yr7fdv1x6x9zfue">https://workbench.4xphq.arvadosapi.com/jobs/4xphq-8i9sb-yr7fdv1x6x9zfue</a></p>
<p>links to</p>
<p><a class="external" href="https://workbench.4xphq.arvadosapi.com/collections/650e9bee8e8b97cb222a6933bd88e777+42%0A">https://workbench.4xphq.arvadosapi.com/collections/650e9bee8e8b97cb222a6933bd88e777+42%0A</a></p>
<p>Even if you remove the %0A,</p>
<p><a class="external" href="https://workbench.4xphq.arvadosapi.com/collections/650e9bee8e8b97cb222a6933bd88e777+42">https://workbench.4xphq.arvadosapi.com/collections/650e9bee8e8b97cb222a6933bd88e777+42</a><br /><a class="external" href="https://workbench.4xphq.arvadosapi.com/collections/650e9bee8e8b97cb222a6933bd88e777">https://workbench.4xphq.arvadosapi.com/collections/650e9bee8e8b97cb222a6933bd88e777</a></p>
<p>give</p>
<p>bad <abbr title="is not URI?">URI</abbr>: <a class="external" href="https://4xphq.arvadosapi.com/arvados/v1/collections/650e9bee8e8b97cb222a6933bd88e777+42">https://4xphq.arvadosapi.com/arvados/v1/collections/650e9bee8e8b97cb222a6933bd88e777+42</a> added 2014-05-07 16:49:32 +0000</p> Arvados - Idea #2752 (Resolved): arv-put can quickly resume an interrupted transfer.https://dev.arvados.org/issues/27522014-05-07T11:47:19ZTom Cleggtom@curii.com
<p>Two main approaches are available. (Eventually we want both. For now, we want whichever is most accessible/efficient from a development perspective.)</p>
HEAD-before-PUT mode
<ul>
<li>Simple to implement reliably</li>
<li>Depends on HEAD actually working (might not be true yet when proxy use case is otherwise ready to go)</li>
<li>Does not depend on local filesystem features like inode/ctime</li>
<li>Does not depend on arv-put running in the same user account (or even host) each time</li>
<li>Still requires re-reading the local data</li>
<li>Interacts interestingly with Keep permission mechanism (needs some combination of storing partial manifests and caching permission signatures locally)</li>
</ul>
Local checkpoint
<ul>
<li>Save state (in <code>$HOME/.cache</code>?) while running.
<ul>
<li>Separate cache per arvados_api_host (don't get confused when uploading to two different sites)</li>
<li>Be attentive to race conditions (e.g., refuse to run two resumable transfers that would use the same cached data)</li>
<li>List of files in order written</li>
<li>For each file (at minimum) store name, inode, ctime, size</li>
<li>List of blobs successfully written to Keep (including size)</li>
</ul>
</li>
<li>When resuming, skip what's already done (unless <code>--no-resume</code> is given).
<ul>
<li>If blob locators have permission signatures, check their expiry times before deciding to re-use. Warn the user (once per arv-put) if blobs are being re-uploaded for this reason.</li>
</ul></li>
</ul> Arvados - Task #2565 (Closed): Check out 2028 notes and sync up with Tomhttps://dev.arvados.org/issues/25652014-04-09T00:17:59ZTom Cleggtom@curii.comArvados - Feature #2375 (Resolved): Add a row to the Logs table whenever a row in any other [inte...https://dev.arvados.org/issues/23752014-03-19T13:52:05ZTom Cleggtom@curii.com
<ul>
<li>Log object is added even if "update" does not actually change the database content (e.g., "node is still alive")</li>
<li>Log object's "properties" attribute <em>(rename it, it's currently called "info")</em> includes:
<ul>
<li>uuid of user making the change (usually same as modified_by, except delete and some updates)</li>
<li>etag of the new version (nil for delete)</li>
<li>etag of the old version (nil for create)</li>
<li>all attributes of the new version (nil if deleting)</li>
<li>all attributes of the old version (nil if creating)</li>
</ul>
</li>
<li>modified_by is current_user (the user whose transaction is being logged)</li>
<li>owner_uuid is system_user (this prevents users from deleting logs)</li>
</ul>