https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422018-01-26T16:53:00ZArvadosArvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=597202018-01-26T16:53:00ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>This is implemented in arvados/services/api/lib</p>
<pre>
when 'is_a'
operand = [operand] unless operand.is_a? Array
cond = []
operand.each do |op|
cl = ArvadosModel::kind_class op
if cl
cond << "#{ar_table_name}.#{attr} like ?"
param_out << cl.uuid_like_pattern
else
cond << "1=0"
end
end
cond_out << cond.join(' OR ')
</pre>
<p>Instead of using 'like' this should use a substring match:</p>
<pre>
operand.each do |op|
cl = ArvadosModel::kind_class op
if cl
cond << "substring(#{ar_table_name}.#{attr}, 6, 5) = ?"
param_out << cl.uuid_prefix
else
cond << "1=0"
end
end
</pre>
<p>If 'attr' is 'uuid' I think you can even short circuit the check with "model_class.uuid_prefix == cl.uuid_prefix" (we can assume every value in the uuid column has the same type.)</p>
<p>To ensure the substring match is efficient, we may need to create an expression index:</p>
<p><a class="external" href="https://www.postgresql.org/docs/9.4/static/indexes-expressional.html">https://www.postgresql.org/docs/9.4/static/indexes-expressional.html</a></p> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=598152018-02-01T20:36:30ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Assigned To</strong> deleted (<del><i>Lucas Di Pentima</i></del>)</li></ul> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=598452018-02-01T21:07:39ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> set to <i>To Be Groomed</i></li></ul> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=599912018-02-07T18:12:17ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Story points</strong> set to <i>1.0</i></li></ul> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=619382018-04-18T15:42:06ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> changed from <i>To Be Groomed</i> to <i>Arvados Future Sprints</i></li></ul> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=697592018-12-12T17:09:28ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2018-12-21 Sprint</i></li></ul> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=697722018-12-12T19:42:10ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=697952018-12-13T16:38:29ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Lucas Di Pentima</i></li></ul> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=698132018-12-13T19:26:43ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="13006: Adds expression index on links.[tail|head]_uuid Arvados-DCO-1.1-Signed-off-by: Lucas Di P..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/cd7a746df5e9bf8a5770d06410b3fe1908282a7b">cd7a746df</a> - branch <code>13006-api-is_a-filter</code><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/1009/">https://ci.curoverse.com/job/developer-run-tests/1009/</a></p>
<ul>
<li>Followed Peter's suggestion by changing the query to use substring(), avoiding the check when <code>attr</code> is "uuid" and prefix_uuid matches.</li>
<li>Added expression index on <code>links.[tail|head]_uuid</code></li>
<li>Added test</li>
</ul>
<p>I also could update the <code>group-sync</code> tool to use the <code>is_a</code> operator again, for enhanced performance.</p> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=698252018-12-13T20:04:39ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<p>Updates at <a class="changeset" title="13006: Adds expression index on links.[tail|head]_uuid Arvados-DCO-1.1-Signed-off-by: Lucas Di P..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/cd7a746df5e9bf8a5770d06410b3fe1908282a7b">cd7a746df</a> - branch <code>13006-api-is_a-filter</code><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/1009/">https://ci.curoverse.com/job/developer-run-tests/1009/</a></p>
<ul>
<li>Followed Peter's suggestion by changing the query to use substring(), avoiding the check when <code>attr</code> is "uuid" and prefix_uuid matches.</li>
</ul>
</blockquote>
<p>If attr == 'uuid' but <code>model_class.uuid_prefix != cl.uuid_prefix</code>, should it fail unconditionally?</p>
<blockquote>
<ul>
<li>Added expression index on <code>links.[tail|head]_uuid</code></li>
<li>Added test</li>
</ul>
<p>I also could update the <code>group-sync</code> tool to use the <code>is_a</code> operator again, for enhanced performance.</p>
</blockquote>
<p>How about in a separate branch.</p> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=698292018-12-13T20:57:35ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote>
<p>If attr == 'uuid' but <code>model_class.uuid_prefix != cl.uuid_prefix</code>, should it fail unconditionally?</p>
</blockquote>
<p>I believe so, fixed.</p>
<p>Update at <a class="changeset" title="13006: Fail when using is_a filter on uuid attr and prefix_uuid doesn't match. Arvados-DCO-1.1-S..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/96165e8fcea44632979ddf7db02c1b70cb95d0f4">96165e8fc</a><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/1011/">https://ci.curoverse.com/job/developer-run-tests/1011/</a></p> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=698302018-12-13T21:05:42ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Peter Amstutz wrote:</p>
<blockquote><blockquote>
<p>I also could update the <code>group-sync</code> tool to use the <code>is_a</code> operator again, for enhanced performance.</p>
</blockquote>
<p>How about in a separate branch.</p>
</blockquote>
<p>Done: <a class="changeset" title="13006: Uses 'is_a' filter again, as it's now able to detect remote objects. Arvados-DCO-1.1-Sign..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/84dcca64f652da1e65bdee67dc171dbd2db361b8">84dcca64f</a> - branch <code>13006-sync-groups-is_a-filter</code><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/1012/">https://ci.curoverse.com/job/developer-run-tests/1012/</a> (not yet started atm)</p> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=698752018-12-14T15:22:57ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<p>Peter Amstutz wrote:</p>
<blockquote><blockquote>
<p>I also could update the <code>group-sync</code> tool to use the <code>is_a</code> operator again, for enhanced performance.</p>
</blockquote>
<p>How about in a separate branch.</p>
</blockquote>
<p>Done: <a class="changeset" title="13006: Uses 'is_a' filter again, as it's now able to detect remote objects. Arvados-DCO-1.1-Sign..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/84dcca64f652da1e65bdee67dc171dbd2db361b8">84dcca64f</a> - branch <code>13006-sync-groups-is_a-filter</code><br />Test run: <a class="external" href="https://ci.curoverse.com/job/developer-run-tests/1012/">https://ci.curoverse.com/job/developer-run-tests/1012/</a> (not yet started atm)</p>
</blockquote>
<p>Is there a 'sync-groups' test that fails without the '13006-api-is_a-filter' fix but passes with it?</p> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=698762018-12-14T15:28:12ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Yes. Although I didn't tested it now, but it was the means to detect this <code>is_a</code> bug in the API server. The test is <code>TestMembershipRemoval</code>, if you want I can rebase this branch to master and then merge the other 13006 branch to prove that the test passes.</p> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=698772018-12-14T15:32:57ZPeter Amstutzpeter.amstutz@curii.com
<ul></ul><p>Lucas Di Pentima wrote:</p>
<blockquote>
<p>Yes. Although I didn't tested it now, but it was the means to detect this <code>is_a</code> bug in the API server. The test is <code>TestMembershipRemoval</code>, if you want I can rebase this branch to master and then merge the other 13006 branch to prove that the test passes.</p>
</blockquote>
<p>Yes, I think that would be a good idea. Go ahead and merge both branches if it goes as expected (test fails without fix → test passes with fix)</p> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=698812018-12-14T15:57:06ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Ok, so I rebased <code>13006-sync-groups-is_a-filter</code> (<a class="changeset" title="13006: Uses 'is_a' filter again because it's being fixed on the API server. Arvados-DCO-1.1-Sign..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/0c21d2bd2d0fd351a4e546002a249b0b748061eb">0c21d2bd2d0fd351a4e546002a249b0b748061eb</a>) to master. Ran the <code>sync-groups</code> tests locally and got the failure:</p>
<pre>
FAIL: sync-groups_test.go:267: TestSuite.TestMembershipRemoval
sync-groups_test.go:308:
c.Assert(GroupMembershipExists(s.cfg.Client, remoteUserUUID, groupUUID), Equals, false)
... obtained bool = true
... expected bool = false
</pre>
<p>Then, merged <code>13006-api-is_a-filter</code> into <code>13006-sync-groups-is_a-filter</code> (<a class="changeset" title="Merge branch '13006-api-is_a-filter' into 13006-sync-groups-is_a-filter Arvados-DCO-1.1-Signed-o..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/cbf93e8d897448dbd52369afe89fef2392140ff1">cbf93e8d897448dbd52369afe89fef2392140ff1</a>). Tests now pass.</p> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=698822018-12-14T16:17:36ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '13006-sync-groups-is_a-filter' Closes #13006 Arvados-DCO-1.1-Signed-off-by: Lucas ..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/cd4bb0b5bde62d7fe32cc105d6311dd318ebe304">arvados|cd4bb0b5bde62d7fe32cc105d6311dd318ebe304</a>.</p> Arvados - Bug #13006: is_a filter doesn't work with remote objectshttps://dev.arvados.org/issues/13006?journal_id=718932019-03-01T18:30:16ZTom Morristfmorris@veritasgenetics.com
<ul><li><strong>Release</strong> set to <i>15</i></li></ul>