Project

General

Profile

Actions

Bug #17507

closed

[keep-web] [s3] Implement NextContinuationToken per ListObjectsV2 API

Added by Tom Clegg about 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Story points:
-
Release relationship:
Auto

Description

According to https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html the ListObjectsV2 API should return NextContinuationToken when isTruncated is true, and accept that value as the continuation-token parameter in a subsequent request.

Currently, keep-web only implements ListObjectsV1 paging (i.e., it sends NextMarker and accepts that value as a marker parameter).

To fix this we should look for the list-type=2 parameter, and if it's set, send NextContinuationToken and accept continuation-token, instead of sending NextMarker and accepting marker.

Probably best to use a base64-encoded filename as a token, rather than the plain filename, in case client implementations rely on the assumption (which appears to be true with AWS) that the token won't have special characters.


Subtasks 1 (0 open1 closed)

Task #17569: Review 17507-listobjectsv2ResolvedTom Clegg04/23/2021Actions

Related issues

Related to Arvados Epics - Idea #16360: Keep-web supports S3 compatible interfaceResolved07/01/202004/30/2021Actions
Actions

Also available in: Atom PDF