https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422021-08-09T17:45:56ZArvadosArvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=959972021-08-09T17:45:56ZWard Vandewegeward@curii.com
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/95997/diff?detail_id=92673">diff</a>)</li><li><strong>Subject</strong> changed from <i>[api] storage class fields should be supported in our filters</i> to <i>[api] storage class fields should be supported in filters</i></li></ul> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=959992021-08-09T17:47:34ZWard Vandewegeward@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-1 priority-4 priority-default" href="/issues/17993">Feature #17993</a>: [deduplication-report] supports storage classes</i> added</li></ul> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=962522021-08-17T20:06:35ZWard Vandewegeward@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed" href="/issues/17697">Idea #17697</a>: Design for reporting tools to determine what data is on multiple storage classes.</i> added</li></ul> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=962572021-08-17T20:07:44ZWard Vandewegeward@curii.com
<ul><li><strong>Blocks</strong> <i><a class="issue tracker-6 status-3 priority-4 priority-default closed behind-schedule" href="/issues/16107">Idea #16107</a>: Storage classes</i> added</li></ul> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=963362021-08-18T15:30:05ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> set to <i>2021-09-01 sprint</i></li><li><strong>Assigned To</strong> set to <i>Tom Clegg</i></li></ul> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=964002021-08-23T18:08:52ZTom Cleggtom@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=964022021-08-23T18:58:32ZTom Cleggtom@curii.com
<ul></ul><p>17994-filter-by-storage-classes @ <a class="changeset" title="17994: Allow filtering on storage_classes_desired/confirmed fields. Arvados-DCO-1.1-Signed-off-b..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/902f8cd258a8dfec749a7f94d478a4027e319750">902f8cd258a8dfec749a7f94d478a4027e319750</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2649/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2649/">developer-run-tests: #2649 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2649" alt="" /></a></a></p>
<p>So far this is a minimal implementation, it accepts filters like <code>[["storage_classes_desired", "=", "[\"default\"]"]]</code> -- note the operand is the JSON representation, as it's stored in the database.</p>
But we probably want these, too:
<ul>
<li><code>[["storage_classes_desired", "=", ["default"]]]</code> (alternative syntax equivalent to <code>"[\"default\"]"</code>)</li>
<li><code>[["storage_classes_desired", "contains", ["default"]]]</code> (matches <code>["foo","default"]</code> as well as exact match <code>["default"]</code>)</li>
</ul>
<p>Currently, <a class="external" href="https://doc.arvados.org/main/api/methods.html">https://doc.arvados.org/main/api/methods.html</a> uses <code>["foo", "contains", "bar"]</code> as the example for "contains", which is a bit misleading since "contains" only works if the first element is "attr.key" where "attr" is a jsonb object column and "key" is a top-level key in the json object. (Should change "foo" to "properties.foo" to make it more clear, I think.)</p>
I'm thinking we could extend that to match json objects/arrays at the top level too, so
<ul>
<li><code>["properties", "contains", ["foo", "bar"]]</code> matches a record with <code>{"foo": 1, "bar": 2, "baz": 3}</code> </li>
<li><code>["storage_classes_desired", "contains", ["foo", "bar"]]</code> matches a record with <code>["bar", "foo", "default"]</code>.</li>
</ul>
<p>The storage_classes_* fields aren't indexed. Practically speaking this might be okay -- there are typically very few classes with lots of collections in each, and if a condition matches a large portion of the table, an index doesn't save much time.</p> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=964032021-08-23T20:15:42ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Related to</strong> <i><a class="issue tracker-2 status-3 priority-4 priority-default closed parent" href="/issues/17995">Feature #17995</a>: [api] add method to get collections where replication_confirmed < replication_desired</i> added</li></ul> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=964112021-08-24T14:10:08ZTom Cleggtom@curii.com
<ul></ul><p>17994-filter-by-storage-classes @ <a class="changeset" title="17994: Update comment. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii.com>" href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/402e69f6e55dce4e11d354c3ca708b8e536c124b">402e69f6e55dce4e11d354c3ca708b8e536c124b</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2651/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2651/">developer-run-tests: #2651 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2651" alt="" /></a></a></p>
<ul>
<li>accepts <code>["storage_classes_confirmed", "contains", ["key1", "key2", ...]]</code> (works on any jsonb column)</li>
<li>accepts <code>["storage_classes_confirmed", "contains", "key1"]</code></li>
<li>reverts adding storage_classes_* to searchable_attributes on collection model (this caused "any" to try to match those columns, which seems undesirable and would require migrating the huge multi-column table index)</li>
<li>accepts "=", "<>", "!=" operators on jsonb columns even if they aren't in searchable_attributes. This makes it possible to do exact matches on storage_classes_*, which could be useful for degenerate cases like a single-element array or an empty properties object.</li>
</ul>
<p>If this seems like the right behavior I'll need to update the API methods docs.</p> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=965222021-08-27T15:23:38ZTom Cleggtom@curii.com
<ul></ul><p>17994-filter-by-storage-classes @ <a class="changeset" title="17994: Document new filtering capabilities. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii...." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/be900941bb4ab286cbeb02f65509be938726d67e">be900941bb4ab286cbeb02f65509be938726d67e</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2662/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2662/">developer-run-tests: #2662 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2662" alt="" /></a></a></p> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=965392021-08-30T16:43:08ZWard Vandewegeward@curii.com
<ul></ul><p>Tom Clegg wrote:</p>
<blockquote>
<p>17994-filter-by-storage-classes @ <a class="changeset" title="17994: Document new filtering capabilities. Arvados-DCO-1.1-Signed-off-by: Tom Clegg <tom@curii...." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/be900941bb4ab286cbeb02f65509be938726d67e">be900941bb4ab286cbeb02f65509be938726d67e</a> -- <a class="external" href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2662/"<a href="https://ci.arvados.org/view/Developer/job/developer-run-tests/2662/">developer-run-tests: #2662 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests&build=2662" alt="" /></a></a></p>
</blockquote>
<p>The developer-run-tests-apps-workbench-integration tests failed so I kicked those off again at <a class="external" href="https://ci.arvados.org/job/developer-run-tests-apps-workbench-integration/2823/console"<a href="https://ci.arvados.org/job/developer-run-tests-apps-workbench-integration/2823/">developer-run-tests-apps-workbench-integration: #2823 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests-apps-workbench-integration&build=2823" alt="" /></a>/console</a>. That failed again, so once more at <a class="external" href="https://ci.arvados.org/job/developer-run-tests-apps-workbench-integration/2824/console"<a href="https://ci.arvados.org/job/developer-run-tests-apps-workbench-integration/2824/">developer-run-tests-apps-workbench-integration: #2824 <img src="https://ci.arvados.org/buildStatus/icon?job=developer-run-tests-apps-workbench-integration&build=2824" alt="" /></a>/console</a>, which finally passed.</p>
<p>LGTM, thanks!</p> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=965442021-08-30T17:29:58ZTom Cleggtom@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '17994-filter-by-storage-classes' into main closes #17994 Arvados-DCO-1.1-Signed-o..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/b1daec9a928eefbc71d8b7368b148fa7b04bf32d">arvados|b1daec9a928eefbc71d8b7368b148fa7b04bf32d</a>.</p> Arvados - Feature #17994: [api] storage class fields should be supported in filtershttps://dev.arvados.org/issues/17994?journal_id=985852021-11-16T16:30:20ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Release</strong> set to <i>42</i></li></ul>