https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422021-03-10T17:40:35ZArvadosArvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=912112021-03-10T17:40:35ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed behind-schedule" href="/issues/16107">Idea #16107</a>: Storage classes</i> added</li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=912122021-03-10T17:41:30ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/91212/diff?detail_id=87923">diff</a>)</li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=912132021-03-10T17:42:20ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/91213/diff?detail_id=87924">diff</a>)</li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=912782021-03-16T17:11:31ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> set to <i>2021-04-14 sprint</i></li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=915022021-03-30T17:34:38ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Category</strong> set to <i>SDKs</i></li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=915732021-03-31T15:37:49ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Lucas Di Pentima</i></li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=915752021-03-31T15:39:55ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-04-14 sprint</i> to <i>2021-05-12 sprint</i></li><li><strong>Assigned To</strong> deleted (<del><i>Lucas Di Pentima</i></del>)</li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=915862021-03-31T16:49:29ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Blocked by</strong> <i><a class="issue tracker-2 status-3 priority-4 priority-default closed parent" href="/issues/13382">Feature #13382</a>: [keepstore] Write new blocks to appropriate storage class</i> added</li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=921012021-04-21T16:27:11ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-2 status-3 priority-4 priority-default closed parent" href="/issues/17572">Feature #17572</a>: arv-mount understands storage classes</i> added</li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=921032021-04-21T16:32:14ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/92103/diff?detail_id=88775">diff</a>)</li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=922772021-04-28T16:18:47ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Lucas Di Pentima</i></li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=925512021-05-12T14:42:47ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-05-12 sprint</i> to <i>2021-05-26 sprint</i></li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=926762021-05-13T15:21:44ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=930812021-05-26T14:20:35ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2021-05-26 sprint</i> to <i>2021-06-09 sprint</i></li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=932502021-05-31T20:42:39ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Status update:</p>
<p>Updates at <a class="changeset" title="17465: Adds tests for class storage support. Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentima <lu..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/a6419676c073a863232c4656f0602b2d038ec3cd">a641967</a> - branch <code>17465-pysdk-storage-classes-support</code><br />Test run: <a class="external" href="https://ci.arvados.org/job/developer-run-tests/2499/"<a href="https://ci.arvados.org/job/developer-run-tests/2499/">developer-run-tests: #2499 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2499" alt="" /></a></a></p>
<ul>
<li><code>KeepClient.put()</code> supports storage classes.</li>
<li>When talking to a older cluster, it falls back to the previous behavior, only checking for the number of copies.</li>
<li>Added tests.</li>
</ul>
<p>Note: Some of the new tests make <code>KeepWriterQueue</code> to freeze. I'm not quite sure what's going on yet, but I avoided the problem my adding 1 to the number of tries (see line 574 at <code>sdk/python/arvados/keep.py</code>)</p>
<p>Pending: Add storage classes support to <code>arvados.Collection</code></p> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=932522021-06-01T01:06:30ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="17465: Adds tests confirming KeepClient storage support on Collection. Also, made storage_classe..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/aeb5185342b751b6bbbf1e17024d6f17417ffaf2">aeb518534</a><br />Test run: <a class="external" href="https://ci.arvados.org/job/developer-run-tests/2500/"<a href="https://ci.arvados.org/job/developer-run-tests/2500/">developer-run-tests: #2500 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2500" alt="" /></a></a></p>
<ul>
<li>Adds <code>KeepClient</code> storage classes support to <code>Collection</code> class through <code>BlockManager</code>.</li>
<li>Collections being loaded from a manifest locator now also load their <code>storage_classes_desired</code> attribute.</li>
<li>Added tests.</li>
</ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=932572021-06-01T14:54:47ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>While reading through <code>arv-put</code> code to see how to tackle <a class="issue tracker-2 status-3 priority-4 priority-default closed parent" title="Feature: [arv-put] Storage classes (Resolved)" href="https://dev.arvados.org/issues/17351">#17351</a>, I'm realizing I did something wrong in regards to the precedence of the storage classes being used when calling <code>Collection.save()</code> or <code>Collection.save_new()</code> and passing storage classes there: The current proposal disregards those passed arguments if the <code>Collection</code> instance was created with other storage classes (something that can happen when the code is called with those arguments or when loading a <code>Collection</code> from the API with storage classes desired attributes already set).</p>
<p>I'll fix that and update the ticket when ready.</p> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=933182021-06-02T20:43:00ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="17465: Collection.save() adds storage_classes_desired when set on instance. Arvados-DCO-1.1-Sign..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/5ab34436e2c47cebedd81fe6d38bdcbc40d6c373">5ab34436e</a><br />Test run: <a class="external" href="https://ci.arvados.org/job/developer-run-tests/2507/"<a href="https://ci.arvados.org/job/developer-run-tests/2507/">developer-run-tests: #2507 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2507" alt="" /></a></a></p>
<ul>
<li>Makes <code>Collection.save()</code> and <code>Collection.save_new()</code> to set the storage classes desired data as a <code>Collection</code> instance variable, so it can be overwritten if already set at instantiation time.</li>
<li>Makes <code>BlockManager</code> to access storage classes desired data from its <code>Collection</code> instead of having an internal copy of the same information. This will allow to immediately start writing blocks on a different set of classes if requested.</li>
<li>Adds one more test.</li>
</ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=933392021-06-03T18:35:49ZTom Cleggtom@curii.com
<ul></ul><p>I'm not sure this is a great solution, but I think it explains/avoids the deadlock you were hitting, where pending_tries reaches 0 because #copies reaches the target, but there are still unsatisfied classes. With this patch, I found tests pass but generate a lot of noise from a queue.Empty exception in testutil -- ISTR the test strategy involves adding extra mocked keep responses in situations like this.</p>
<pre><code class="diff syntaxhl"><span class="gh">diff --git a/sdk/python/arvados/keep.py b/sdk/python/arvados/keep.py
index 2f20132ae..8fe05f347 100644
</span><span class="gd">--- a/sdk/python/arvados/keep.py
</span><span class="gi">+++ b/sdk/python/arvados/keep.py
</span><span class="p">@@ -571,7 +571,7 @@</span> class KeepClient(object):
self.response = None
self.storage_classes_tracking = True
self.queue_data_lock = threading.Lock()
<span class="gd">- self.pending_tries = max(copies, len(classes))+1
</span><span class="gi">+ self.pending_tries = max(copies, len(classes))
</span> self.pending_tries_notification = threading.Condition()
def write_success(self, response, replicas_nr, classes_confirmed):
<span class="p">@@ -585,6 +585,11 @@</span> class KeepClient(object):
self.confirmed_storage_classes[st_class] += st_copies
except KeyError:
self.confirmed_storage_classes[st_class] = st_copies
<span class="gi">+ pending_classes = 0
+ for st_class in self.wanted_storage_classes:
+ if st_class not in self.confirmed_storage_classes or self.confirmed_storage_classes[st_class] < self.wanted_copies:
+ pending_classes += 1
+ self.pending_tries = max(self.wanted_copies - self.successful_copies, pending_classes)
</span> self.response = response
with self.pending_tries_notification:
self.pending_tries_notification.notify_all()
</code></pre>
<p><a class="source" href="https://dev.arvados.org/projects/arvados/repository/arvados/entry/sdk/python/arvados/collection.py">source:sdk/python/arvados/collection.py</a> L1427 ("classes = ...") seems to be unused/unneeded:</p>
<pre>
if self._block_manager is None:
copies = (self.replication_desired or
self._my_api()._rootDesc.get('defaultCollectionReplication',
2))
classes = self.storage_classes_desired or []
self._block_manager = _BlockManager(self._my_keep(), copies=copies, put_threads=self.put_threads, num_retries=self.num_retries, storage_classes_func=self.storage_classes_desired)
</pre>
<p><a class="source" href="https://dev.arvados.org/projects/arvados/repository/arvados/entry/sdk/python/arvados/keep.py">source:sdk/python/arvados/keep.py</a> L1256 seems like it will print this warning once per block on a big upload. Maybe use an "already warned" flag on the KeepClient object or something like that?</p>
<pre>
_logger.warning("X-Keep-Storage-Classes header not supported by the cluster")
</pre> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=933502021-06-03T19:55:55ZTom Cleggtom@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-3 priority-4 priority-default closed parent" href="/issues/17351">Feature #17351</a>: [arv-put] Storage classes</i> added</li></ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=934042021-06-07T14:13:13ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Thanks for your suggestion, I applied it with some tweaks to avoid code duplication.</p>
<p>Updates at <a class="changeset" title="17465: Fixed tests and made the suite less noisy. Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentim..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/205aecd0875343feaec88e7f0e78dd6e9db1fd9c">205aecd08</a><br />Test run: <a class="external" href="https://ci.arvados.org/job/developer-run-tests/2509/"<a href="https://ci.arvados.org/job/developer-run-tests/2509/">developer-run-tests: #2509 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2509" alt="" /></a></a></p>
<ul>
<li>Applied suggested fix to recompute pending tries.</li>
<li>Avoids a "unsupported feature" warning logging for every block upload.</li>
<li>Fixed tests.</li>
</ul> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=934202021-06-07T17:16:03ZTom Cleggtom@curii.com
<ul></ul><p><a class="changeset" title="17465: Fixed tests and made the suite less noisy. Arvados-DCO-1.1-Signed-off-by: Lucas Di Pentim..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/205aecd0875343feaec88e7f0e78dd6e9db1fd9c">205aecd08</a> LGTM, thanks!</p> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=934232021-06-07T18:01:23ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '17465-pysdk-storage-classes-support' Closes #17465 Arvados-DCO-1.1-Signed-off-by: ..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/8227b8a1be943fbbf3adb23a3d549dec28efbbba">arvados|8227b8a1be943fbbf3adb23a3d549dec28efbbba</a>.</p> Arvados - Idea #17465: Support writing blocks to correct storage classes in Python SDKhttps://dev.arvados.org/issues/17465?journal_id=985512021-11-16T16:26:22ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>42</i></li></ul>