Bug #17522
Updated by Tom Clegg about 3 years ago
When the input filename is "-" or "/dev/stdin", arv-put appears to read/transcode the input as utf-8:
<pre>
tom@shell:~$ head -c1000000 /dev/urandom | arv-put /dev/stdin
2021-04-09 16:54:34 arvados.arv_put[29927] INFO: No cache usage requested for this run.
Traceback (most recent call last):
File "/usr/bin/arv-put", line 7, in <module>
main()
File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/arvados/commands/put.py", line 1270, in main
trash_at=trash_at)
File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/arvados/commands/put.py", line 508, in __init__
self._build_upload_list()
File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/arvados/commands/put.py", line 526, in _build_upload_list
self._write_stdin(self.filename or 'stdin')
File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/arvados/commands/put.py", line 746, in _write_stdin
self._write(sys.stdin, output)
File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/site-packages/arvados/commands/put.py", line 839, in _write
data = source_fd.read(arvados.config.KEEP_BLOCK_SIZE)
File "/usr/share/python3/dist/python3-arvados-python-client/lib/python3.7/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa1 in position 1: invalid start byte
</pre>
As a workaround you can make a symlink so arv-put doesn't realize it's reading from stdin.
<pre>
tom@shell:~$ ln -s /dev/stdin stdin
tom@shell:~$ head -c1000000 /dev/urandom | arv-put ./stdin
2021-04-09 16:56:11 arvados.arv_put[31332] INFO: Creating new cache file at /home/tom/.cache/arvados/arv-put/30973df233a1b57881df8fc58ff569bd
1000000 2021-04-09 16:56:11 arvados.arv_put[31332] INFO:
2021-04-09 16:56:11 arvados.arv_put[31332] INFO: Collection saved as 'Saved at 2021-04-09 16:56:11 UTC by tom@shell.2xpu4.arvadosapi.com'
2xpu4-4zz18-fovltoc3t3r3bu3
</pre>