Project

General

Profile

Actions
Copy link

Bug #3432

closed

Uploading files into docker image using keep

Added by Bryan Cosca over 9 years ago. Updated about 7 years ago.

Status:
Closed
Priority:
Normal
Assigned To:
-
Category:
SDKs
Target version:
-
Story points:
1.0

Description

$ arv-get 2e98fdc8e90f4c48a0714b711767c9ce+76
. 3163cbeef8fd50d8cb85096758b801a3+12404311+A6fa27b2185f6c706df383c4e832c23d23b65c008@53ecda1b 0:12404311:GenomeAnalysisTK.jar

In docker:

root@a9ede8e37919:/# arv-get 2e98fdc8e90f4c48a0714b711767c9ce+76/ .
Traceback (most recent call last):
  File "/usr/local/bin/arv-get", line 197, in <module>
    for data in f.readall():
  File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 155, in readall
    data = self.read(size)
  File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 139, in read
    data += self._stream.readfrom(locator+segmentoffset, segmentsize)
  File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 265, in readfrom
    data += self._keep.get(locator)[segmentoffset:segmentoffset+segmentsize]
  File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 432, in get
    raise arvados.errors.NotFoundError("Block not found: %s" % expect_hash)
arvados.errors.NotFoundError: Block not found: 3163cbeef8fd50d8cb85096758b801a3
root@a9ede8e37919:/# arv-get 2e98fdc8e90f4c48a0714b711767c9ce+76/  
usage: arv-get [-h] [--progress | --no-progress | --batch-progress]
               [--hash HASH | --md5sum] [-n] [-r] [-f | --skip-existing]
               locator [destination]
arv-get: error: Destination is not a directory.

Actions
Copy link
 #1

Updated by Bryan Cosca over 9 years ago

This was within docker. I was able to get the file outside of docker

Actions
Copy link
 #2

Updated by Bryan Cosca over 9 years ago

also my current version was: arvados-python-client==0.1.1406213925.7223231 I got this error,

I did a pip install --upgrade then I get this error:
root@a9ede8e37919:/temp# arv-get 2e98fdc8e90f4c48a0714b711767c9ce+76/ .
Traceback (most recent call last):
File "/usr/local/bin/arv-get", line 197, in <module>
for data in f.readall():
File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 155, in readall
data = self.read(size)
File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 139, in read
data += self._stream.readfrom(locator+segmentoffset, segmentsize)
File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 265, in readfrom
data += self._keep.get(locator)[segmentoffset:segmentoffset+segmentsize]
File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 432, in get
raise arvados.errors.NotFoundError("Block not found: %s" % expect_hash)
arvados.errors.NotFoundError: Block not found: 3163cbeef8fd50d8cb85096758b801a3
root@a9ede8e37919:/temp# ls
GenomeAnalysisTK.jar picard-tools-1.118 picard.zip
root@a9ede8e37919:/temp# md5sum GenomeAnalysisTK.jar
d41d8cd98f00b204e9800998ecf8427e GenomeAnalysisTK.jar
root@a9ede8e37919:/temp# arv-get 2e98fdc8e90f4c48a0714b711767c9ce+76/ .
Traceback (most recent call last):
File "/usr/local/bin/arv-get", line 197, in <module>
for data in f.readall():
File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 155, in readall
data = self.read(size)
File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 139, in read
data += self._stream.readfrom(locator+segmentoffset, segmentsize)
File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 265, in readfrom
data += self._keep.get(locator)[segmentoffset:segmentoffset+segmentsize]
File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 432, in get
raise arvados.errors.NotFoundError("Block not found: %s" % expect_hash)
arvados.errors.NotFoundError: Block not found: 3163cbeef8fd50d8cb85096758b801a3
root@a9ede8e37919:/temp# ls
GenomeAnalysisTK.jar picard-tools-1.118 picard.zip
root@a9ede8e37919:/temp# md5sum GenomeAnalysisTK.jar
d41d8cd98f00b204e9800998ecf8427e GenomeAnalysisTK.jar

:/scratch/temp$ md5sum GenomeAnalysisTK.jar
3163cbeef8fd50d8cb85096758b801a3 GenomeAnalysisTK.jar

BUT, the file is there but the md5sum does not match.

Actions
Copy link
 #3

Updated by Ward Vandewege over 9 years ago

  • Target version set to Bug Triage
Actions
Copy link
 #4

Updated by Ward Vandewege over 9 years ago

  • Project changed from Arvados to 35
Actions
Copy link
 #5

Updated by Tim Pierce over 9 years ago

  • Target version changed from Bug Triage to 2014-10-08 sprint
Actions
Copy link
 #6

Updated by Tim Pierce over 9 years ago

  • Category set to SDKs
  • Assigned To set to Tim Pierce
  • Project changed from 35 to Arvados
Actions
Copy link
 #7

Updated by Tim Pierce over 9 years ago

  • Description updated (diff)
  • Target version changed from 2014-10-08 sprint to Arvados Future Sprints
  • Story points set to 1.0

This problem persists.

bcosc.qr1hi$ /data/twp$ docker run -t -i bcosc/all-tools
root@2264f7a7acba:/# HISTIGNORE=$HISTIGNORE:'export ARVADOS_API_TOKEN=*'
root@2264f7a7acba:/# export ARVADOS_API_TOKEN=**************************************************
root@2264f7a7acba:/# export ARVADOS_API_HOST=qr1hi.arvadosapi.com
root@2264f7a7acba:/# unset ARVADOS_API_HOST_INSECURE
root@2264f7a7acba:/# arv-get 2e98fdc8e90f4c48a0714b711767c9ce+76/ .
Traceback (most recent call last):
  File "/usr/local/bin/arv-get", line 202, in <module>
    for data in f.readall():
  File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 157, in readall
    data = self.read(size)
  File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 140, in read
    data = self._stream.readfrom(locator+segmentoffset, segmentsize)
  File "/usr/local/lib/python2.7/dist-packages/arvados/stream.py", line 266, in readfrom
    data += self._keep.get(locator)[segmentoffset:segmentoffset+segmentsize]
  File "/usr/local/lib/python2.7/dist-packages/arvados/keep.py", line 638, in get
    raise arvados.errors.KeepReadError(loc_s)
arvados.errors.KeepReadError: 3163cbeef8fd50d8cb85096758b801a3+12404311+A1aa7b33e71adc4d198f48245941422ed6fe5b445@54383316

Same result when using the collection uuid qr1hi-4zz18-zh2zqh7aojixlaq.

Suspect a Docker NAT issue. Will investigate further.

Actions
Copy link
 #8

Updated by Tom Morris over 7 years ago

  • Status changed from New to Closed
  • Assigned To deleted (Tim Pierce)

This seems to have gotten lost. It's so old that I'm just going to close it.

Actions
Copy link
 #9

Updated by Tom Clegg about 7 years ago

  • Target version deleted (Arvados Future Sprints)
Actions
Copy link

Also available in: Atom PDF