Arvados

I suspect this isn't caused by a timestamp collision on the last collection.

Data manager is using 1s precision to format its time filters. (This is a bug.)

If the last collection received had modtime 2112-12-21T21:12.211202112Z then our sanity check will count "modified_at <= 2112-12-21T21:12Z" and therefore won't even count the last collection we know about. So it seems the sanity check is being less conservative than we want, and you're really missing 3 collections somewhere, not 2.

Looking for other explanations, it seems that our use of "offset" in this loop is unsafe.

Say b1, b2, b3 have the same modtime (to within one second, because datamanager is filtering dates with 1s precision) and page size is 2.

collections at start    a, b1, b2, b3, c, d, e, f, g, h, i, j
offset=0                a, b1
modtime>=b1, offset=0      b1, b2
                              (b2 is modified now, and therefore becomes "k")
collections database    a, b1,     b3, c, d, e, f, g, h, i, j, k
modtime>=b1, offset=2                  c, d
modtime>=c, offset=0                         e, f

(Oops, never saw b3)

Could this be responsible?

I agree the lack of precision in data manager timestamps is a bug, but this final pass through the loop would apply with more precision as well if it is theoretically possible for two collections to have identical high precision timestamps. If the API server or database guarantees that no two collections can have the same high precision timestamp, then this would not be an issue.

If the API server interprets "modified_at <= 2112-12-21T21:12Z" as being equal to "modified_at <= 2112-12-21T21:12.000000000Z", such that it would not match "2112-12-21T21:12.211202112Z", then I agree that is another potential explanation for the errors I observed, and this fix would not solve it.

This fix was assuming that truncated timestamps would be compared only in as much precision as they specify (i.e. it would assume that a query for "modified_at 2112-12-21T21:12Z" would match "2112-12-21T21:12.211202112Z" because they are equal up to the second). If that isn't true, then this fix is pretty much pointless. Obviously if accessing postgres directly, it wouldn't match unless you date_trunc the timestamp column, but it seems reasonable for the API server to implement '' comparisons with truncation such that you could do a query like "modified_at == 2112-12-21" and it would automatically truncate the modified_at column to the appropriate precision to make the comparison make sense. If it doesn't do that, it should probably be an error to specify comparisons against timestamps that are not specified to the full precision available.