Project

General

Profile

Actions

Idea #16360

closed

Keep-web supports S3 compatible interface

Added by Peter Amstutz over 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
-
Target version:
-
Start date:
07/01/2020
Due date:
04/30/2021
Story points:
-
Release:
Release relationship:
Auto

Description

Background: Applications want to access data in Arvados through a web API. Keep-web currently supports WebDAV. Although it is a standard, 3rd party applications used by customers do not support WebDAV. Instead, many 3rd party applications support reading and writing to S3-compatible object storage. Keep-web should support an S3 compatible API to enable these applications to easily access Arvados.

Initial implementation should support the "list objects", "get object", and "put object" APIs using
  • object name = file path within collection
  • bucket name = collection UUID
  • secret key = Arvados API token
  • endpoint = ExternalURL of keep-web service

...with enough s3 compatibility to test with a common s3 client library like https://github.com/minio/minio-go.


Related issues 10 (2 open8 closed)

Related to Arvados - Idea #16535: [keep-web] Minimal implementation of S3 APIResolvedTom Clegg07/21/2020Actions
Related to Arvados - Feature #16744: [keep-web] Support more S3 write APIs: DeleteObjects, POST objectNewActions
Related to Arvados - Idea #16809: [keep-web] Check V4 signature on S3 requests, don't require sending entire Arvados token as AccessKeyResolvedTom Clegg09/22/2020Actions
Related to Arvados - Bug #16830: [keep-web] S3 PutObject response should have content MD5NewActions
Related to Arvados - Bug #16850: [keep-web] Add KeyCount field to ListObjects responseResolvedTom Clegg09/21/2020Actions
Related to Arvados - Bug #16790: [keep-web] S3 ListObjects response should not have empty NextMarker fieldResolvedTom Clegg09/01/2020Actions
Related to Arvados - Feature #17009: [keep-web] S3 API should accept bucket name as first component of domain nameResolvedTom Clegg11/19/2020Actions
Related to Arvados - Feature #17119: Virtual folder in FUSE/S3/WebDAV with contents defined by a queryResolvedWard Vandewege02/12/2021Actions
Related to Arvados - Feature #16669: Accept OpenID Connect access tokenResolvedTom Clegg09/24/2020Actions
Related to Arvados - Bug #17507: [keep-web] [s3] Implement NextContinuationToken per ListObjectsV2 APIResolvedTom Clegg04/23/2021Actions
Actions

Also available in: Atom PDF