https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422014-12-10T19:09:25ZArvadosArvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=190482014-12-10T19:09:25ZTom Cleggtom@curii.com
<ul><li><strong>Story points</strong> set to <i>0.5</i></li></ul> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=190492014-12-10T19:09:38ZTom Cleggtom@curii.com
<ul><li><strong>Subject</strong> changed from <i>greater than filter is treated as greater than or equals for modification time.</i> to <i>[API] greater than filter is treated as greater than or equals for modification time.</i></li></ul> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=190502014-12-10T19:09:45ZTom Cleggtom@curii.com
<ul><li><strong>Target version</strong> set to <i>Arvados Future Sprints</i></li></ul> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=197752015-01-07T20:30:54ZTom Cleggtom@curii.com
<ul><li><strong>Target version</strong> changed from <i>Arvados Future Sprints</i> to <i>2015-01-28 Sprint</i></li></ul> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=197952015-01-07T20:44:25ZRadhika Chippadaradhika@curoverse.com
<ul><li><strong>Assigned To</strong> set to <i>Radhika Chippada</i></li></ul> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=206802015-01-29T17:54:42ZTom Cleggtom@curii.com
<ul><li><strong>Target version</strong> changed from <i>2015-01-28 Sprint</i> to <i>2015-02-18 sprint</i></li></ul> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=211022015-02-12T02:55:05ZRadhika Chippadaradhika@curoverse.com
<ul></ul><ul>
<li>We do have more precision in our tables than we are returning. Below is an example row from my db<br />whose modified_at time is <strong>2015-02-11 20:59:41.349813</strong> and API returned value is <strong>2015-02-11T20:59:41Z</strong>.<br />Details below. Do we want to include milliseconds also to what we return in our API calls?</li>
</ul>
<ul>
<li>Row in my db at <strong>2015-02-11 20:59:41.349813</strong></li>
</ul>
<pre>
arvados_dev=# select modified_at from collections where uuid='<a href="https://arvadosapi.com/zzzzz-4zz18-0f5ltxrlt7ljkwr">zzzzz-4zz18-0f5ltxrlt7ljkwr</a>';
modified_at
----------------------------
2015-02-11 20:59:41.349813
(1 row)
</pre>
<ul>
<li>When I search with filters=[['modified_at','>','2015-02-11T20:59:41Z']], I get this collection and the<br />timestamps returned are only up to the precision of seconds <strong>u'modified_at': u'2015-02-11T20:59:41Z'</strong></li>
</ul>
<pre>
python -c "import arvados; import pprint; pprint.pprint(arvados.api().collections().list
(order='modified_at',filters=[['modified_at','>','2015-02-11T20:59:41Z']]).execute())"
{u'etag': u'',
u'items': [{u'created_at': u'2015-02-11T20:59:41Z',
u'description': None,
u'etag': u'2ca065kedhr7agmm2fg09o2l1',
u'href': u'/collections/zzzzz-4zz18-0f5ltxrlt7ljkwr',
u'kind': u'arvados#collection',
u'modified_at': u'2015-02-11T20:59:41Z',
u'modified_by_client_uuid': u'<a href="https://arvadosapi.com/zzzzz-ozdt8-rhk40xi1j0ieseg">zzzzz-ozdt8-rhk40xi1j0ieseg</a>',
u'modified_by_user_uuid': u'<a href="https://arvadosapi.com/zzzzz-tpzed-fosbryt7pvj8qqj">zzzzz-tpzed-fosbryt7pvj8qqj</a>',
u'name': u'Collection created at 2015-02-11 15:59:41 -0500',
u'owner_uuid': u'<a href="https://arvadosapi.com/zzzzz-j7d0g-kk122lkquft16cn">zzzzz-j7d0g-kk122lkquft16cn</a>',
u'portable_data_hash': u'4015d5536d0a89f45f0251dbaf84fcb5+141',
u'properties': None,
u'replication_desired': 2,
u'uuid': u'<a href="https://arvadosapi.com/zzzzz-4zz18-0f5ltxrlt7ljkwr">zzzzz-4zz18-0f5ltxrlt7ljkwr</a>'}],
u'items_available': 1,
u'kind': u'arvados#collectionList',
u'limit': 100,
u'offset': 0,
u'self_link': u''}
</pre>
<ul>
<li>If I search with <strong>filters=[['modified_at','>','2015-02-11T20:59:41.5Z']]</strong>, I get 0 collections.</li>
</ul>
<pre>
python -c "import arvados; import pprint; pprint.pprint(arvados.api().collections().list
(order='modified_at',filters=[['modified_at','>','2015-02-11T20:59:41.5Z']]).execute())"
{u'etag': u'',
u'items': [],
u'items_available': 0,
u'kind': u'arvados#collectionList',
u'limit': 100,
u'offset': 0,
u'self_link': u''}
</pre> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=211552015-02-13T18:56:12ZRadhika Chippadaradhika@curoverse.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul><ul>
<li>Added a getter method for collection modified_at attribute with formatting to include milliseconds in api responses for collection objects.
<ul>
<li>I experimented with en.yml (i18n) and adding a time formats initializer. These did not help.</li>
<li>The getter method however helped in including ms in collection json returned.</li>
</ul></li>
</ul>
<ul>
<li>I saw the milliseconds in api responses returned (using the above python example) as well as when workbench makes a collection request.
<ul>
<li>Observed no side effect in workbench. Workbench seems to wrap the json responses into model objects using discovery document and hence it only sees TimeWithZone (as before).</li>
</ul></li>
</ul>
<ul>
<li>For now, I added the extended timestamp format for collection -> modified_at attributes only.</li>
</ul>
<pre>
python -c "import arvados; import pprint; pprint.pprint(arvados.api().collections().list
(order='modified_at',filters=[['modified_at','>','2015-02-13 15:18:41Z']]).execute())"
{u'etag': u'',
u'items': [{u'created_at': u'2015-02-13T15:18:41Z',
u'description': None,
u'etag': u'4bcxw5sjcfw2p1hugcvwexfyz',
u'href': u'/collections/zzzzz-4zz18-3y0ohzd3mlwyjf3',
u'kind': u'arvados#collection',
u'modified_at': u'2015-02-13T15:18:41.108518000Z',
u'modified_by_client_uuid': u'<a href="https://arvadosapi.com/zzzzz-ozdt8-rhk40xi1j0ieseg">zzzzz-ozdt8-rhk40xi1j0ieseg</a>',
u'modified_by_user_uuid': u'<a href="https://arvadosapi.com/zzzzz-tpzed-fosbryt7pvj8qqj">zzzzz-tpzed-fosbryt7pvj8qqj</a>',
u'name': u'Collection created at 2015-02-13 10:18:40 -0500',
u'owner_uuid': u'<a href="https://arvadosapi.com/zzzzz-j7d0g-kk122lkquft16cn">zzzzz-j7d0g-kk122lkquft16cn</a>',
u'portable_data_hash': u'e5ef2d7589ddd0c8405c6972c17b16fc+53',
u'properties': None,
u'replication_desired': 2,
u'uuid': u'<a href="https://arvadosapi.com/zzzzz-4zz18-3y0ohzd3mlwyjf3">zzzzz-4zz18-3y0ohzd3mlwyjf3</a>'}],
u'items_available': 1,
u'kind': u'arvados#collectionList',
u'limit': 100,
u'offset': 0,
u'self_link': u''}
</pre> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=212192015-02-16T16:42:34ZTom Cleggtom@curii.com
<ul></ul><p>It doesn't seem quite right to make the models' attribute methods start returning strings instead of ActiveSupport::TimeWithZone. I'd like it better if we just changed the formatting without changing where/when times get converted to strings.</p>
<p>I think the ideal solution would change the default time formatter (and the one Oj uses) to our desired format. I haven't found the right hooks for that, but it might be possible (if repetitive) to tell acts_as_api which format we want to use in our API responses -- see <a class="changeset" title="4759: Use ISO 8601 timestamps with fractional seconds in API responses." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/4b79e0e764d79745dc4d9cf2a03c871dbc772027">4b79e0e</a> on 4759-timestamp-precision-TC. (It fails a lot of tests with ActiveModel::MissingAttributeError, though, I think because of the "select" stuff.)</p>
<p>How about something like that + the test case from 4759-timestamp-precision?</p>
Suggestions for the test case:
<ul>
<li>Move to functional tests</li>
<li>Split "get" and "put" into separate tests (or just drop the "put" test -- it uses the same API response, right?)</li>
<li>In the "put" test, change something other than manifest_text, to avoid weighing down the test with manifest signing code</li>
<li>Likewise in "search": better to test using the cheapest/easiest possible path to the code being tested.</li>
<li>Add a trailing Z, and ^ and $ anchors, to the regexp</li>
<li>Write the regexp once as a constant and reuse, instead of copying & pasting</li>
</ul>
Unrelated aside, in existing code:
<ul>
<li>The second argument to <code>search_using_filter search_filter, expected_items</code> seems to be used as a sneaky boolean: if you pass it <code>3</code>, <code>false</code>, or <code>nil</code> it asserts <em>at least 1 result.</em> Should probably treat it as a boolean where falsy means "zero results" and truthy means "non-zero results"...</li>
</ul> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=212282015-02-16T19:02:25ZTom Cleggtom@curii.com
<ul></ul><p>Replaced 4759-timestamp-precision-TC with much nicer <a class="changeset" title="4759: Use ISO 8601 timestamps with fractional seconds in API responses." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/5f2d5f96abb8941bed95bd9f47f79f6f1c3ba38d">5f2d5f9</a>.</p> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=212502015-02-17T06:41:17ZTom Cleggtom@curii.com
<ul></ul><p>4759-timestamp-precision-TC now has</p>
<p><a class="changeset" title="4759: Add functional tests for timestamp precision." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/5517f022cdb6233551c9281422c033d18293ec03">5517f02</a> 4759: Add functional tests for timestamp precision.<br /><a class="changeset" title="4759: Add test for inequality filters." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/6ee389b798bd7898e16b8ab8bf9394bf97c40f46">6ee389b</a> 4759: Add test for inequality filters.<br /><a class="changeset" title="4759: Ignore args to as_json." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/4f8b2d755cb8249cd9118b7d3213a0021b83b0cb">4f8b2d7</a> 4759: Ignore args to as_json.<br /><a class="changeset" title="4759: Use ISO 8601 timestamps with fractional seconds in API responses." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/5f2d5f96abb8941bed95bd9f47f79f6f1c3ba38d">5f2d5f9</a> 4759: Use ISO 8601 timestamps with fractional seconds in API responses.</p>
<a name="Behavior"></a>
<h3 >Behavior<a href="#Behavior" class="wiki-anchor">¶</a></h3>
I think <a class="changeset" title="4759: Add test for inequality filters." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/6ee389b798bd7898e16b8ab8bf9394bf97c40f46">6ee389b</a> is the best test of what we really want: the timestamps <em>provided by the API server</em> can be used in inequality filters with the expected results. If a list/get/whatever query returned a record R with <code>"modified_at":"X"</code>, then:
<ul>
<li>Filter <code>modified_at>="X"</code> should include R;</li>
<li>Filter <code>modified_at>"X"</code> should not include R.</li>
</ul>
<p>If the <em>client</em> truncates the precision (to 1 second, or 1 day, or whatever) we'll still interpret that as an earlier time rather than a range. I think this is fine. (Arguably it would be reasonable to interpret <code>[["modified_at","=","2015-02-17"]]</code> as <code>[["modified_at",">=","2015-02-17T00:00:00.000000000Z"],["modified_at","<","2015-02-18T00:00:00.000000000Z"]]</code> ...but that sounds more like a feature than a bugfix, the use case isn't obvious, and a client could accomplish the same effect easily enough (and more generally -- not just on zero boundaries) without our help, so it doesn't seem especially urgent.)</p>
So the two examples in <a class="issue tracker-1 status-3 priority-4 priority-default closed parent" title="Bug: [API] greater than filter is treated as greater than or equals for modification time. (Resolved)" href="https://dev.arvados.org/issues/4759#note-7">#4759#note-7</a> note-7 should (continue to) work as shown:
<ul>
<li>The first has a literal timestamp with no fractional second, which can be taken to mean ...:41.000000000, so a record with timestamp ...:41.349813 is in fact later.</li>
<li>The second asks for timestamps later than ...:41.5, which ...:41.349813 is not.</li>
</ul> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=212942015-02-17T16:37:00ZRadhika Chippadaradhika@curoverse.com
<ul></ul><p>Tom, yes, I agree that "2015-02-17" should mean "2015-02-17T00:00:00.000000000Z" and all the updates look good.</p>
<p>One minor comment about the tests. In the python test_timestamp_inequality_filter, why use specimen instead of any other object such as collection. We had talked about removing humans, specimens etc before and I think it would be nice to not use these object types in new tests.</p> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=212982015-02-17T17:04:41ZTom Cleggtom@curii.com
<ul></ul><p>Radhika Chippada wrote:</p>
<blockquote>
<p>One minor comment about the tests. In the python test_timestamp_inequality_filter, why use specimen instead of any other object such as collection. We had talked about removing humans, specimens etc before and I think it would be nice to not use these object types in new tests.</p>
</blockquote>
<p>I tend to use specimen to isolate tests from special behavior like Collection's manifest_text signatures and portable_data_hash: It's just a generic Arvados type with features like uuid and modified_at. (If we do decide to remove Specimen, will we replace it with some other type?)</p> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=212992015-02-17T17:07:09ZRadhika Chippadaradhika@curoverse.com
<ul></ul><p>Tom, that is fine. I also used "humans" in several tests earlier for the same reason. We can deal with them when we delete these data models from our system.</p>
<p>LGTM</p> Arvados - Bug #4759: [API] greater than filter is treated as greater than or equals for modification time.https://dev.arvados.org/issues/4759?journal_id=213012015-02-17T17:35:08ZAnonymous
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>100</i></li></ul><p>Applied in changeset arvados|commit:3598c3003a7987cca5c0536ba8206ec40c1c3649.</p>