Arvados

Symptom:¶

The behavior of arv-put is inconsistent depending on whether it is talking to the keepstore service directly or via keep-proxy. When it is using keepstore (for example, when running on the same host as the server), running arv-put [input_files...] command generates a spurious message like the following:

2024-07-08 13:45:14 arvados.keep[10196] WARNING: X-Keep-Storage-Classes header not supported by the cluster

Running the same command on a host using keep-proxy will not trigger this problem.

The expected behavior is that arv-put should behave consistently no matter interfacing with keepstore or keep-proxy, and in particular it should not erroneously complain about missing storage-class support.

Cause:¶

This issue stems from how keepstore and keep-proxy generate the X-Keep-Storage-Classes-Confirmed (and possibly other related) HTTP header differently.

Keep-proxy writes the header's value as a comma-separated string of substrings in the key=value format, e.g.

X-Keep-Storage-Classes-Confirmed: default=1, hot=1

See: source:services/keepproxy/keepproxy.go@d9592ca6a1a9da3adb5c7a5b8201765dad3af93b#L506

However, keepstore decides that the separator should be a semicolon:

x-keep-storage-classes-confirmed: default=1; hot=1

See: source:services/keepstore/router.go@d9592ca6a1a9da3adb5c7a5b8201765dad3af93b#L131

Therefore the arv-put client can't parse the output of keepstore and decides that the header is as good as missing; see: source:sdk/python/arvados/keep.py@d9592ca6a1a9da3adb5c7a5b8201765dad3af93b#L814

Suggested solution:¶

Settle on the comma, based on the HTTP semantics specification (RFC 9110). In this way, the client program doesn't need any modification.