Bug #9233
closed
Fix Ancestry Mapper pipeline
Description
- Update its components on qr1hi so they complete successfully on the default input
- Copy the pipeline to https://workbench.qr1hi.arvadosapi.com/projects/qr1hi-j7d0g-662ij1pcw6bj8uj
Currently, the pipeline fails on SNP-Annotation-justmap (missing an input). All the failures so far have been due to outdated run-command problems, so this story is not expected to take a long time, even though the work is not completely determined.
Latest run of pipeline: https://workbench.qr1hi.arvadosapi.com/pipeline_instances/qr1hi-d1hrv-q75kebcrxhex370
This story is considered complete when it completes successfully on its default inputs and the updated pipeline is in the Public Project.
Updated by Bryan Cosca over 8 years ago
Updated a bunch of packages on the docker image and ran into this:
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr File "/tmp/crunch-job/src/crunch_scripts/run-command", line 42, in <module> 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr t = arvados.current_task().tmpdir 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/__init__.py", line 65, in current_task 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr task = api('v1').job_tasks().get(uuid=os.environ['TASK_UUID']).execute() 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 135, in positional_wrapper 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr return wrapped(*args, **kwargs) 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 755, in execute 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr method=str(self.method), body=self.body, headers=self.headers) 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 93, in _retry_request 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr resp, content = http.request(uri, method, *args, **kwargs) 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr TypeError: _intercept_http_request() takes exactly 2 arguments (5 given) 2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr srun: error: compute4: task 0: Exited with exit code 1
Updated by Bryan Cosca over 8 years ago
I'm not specifying arvados_sdk_version in the template, and am running into this error still: instance: qr1hi-d1hrv-n3f1ow28sdj58qr job: qr1hi-8i9sb-mo15okvl037dknp
"RemoveIndels-fixit": {
"script": "run-command",
"script_version": "79509812aefb7c32cb7755adc77a716b8491dec8",
"repository": "bryan.coscamasonancestry",
"script_parameters": {
"command": {
"value": [
"bash",
"-c",
"(sed 's/chr//g' $(glob $(dir $(input))/*.vcf)) > $(basename $(glob $(dir $(input))/*.vcf)).fixit.vcf"
]
},
"input": {
"required": true,
"dataclass": "Collection",
"title": "Input VCF file + 00-all.vcf + 1KG VCFs",
"default": "87673c2b7662affafc8657ca1ebf7c8c+98212",
"value": "87673c2b7662affafc8657ca1ebf7c8c+98212"
},
"task.vwd": {
"value": "$(input)"
}
},
"runtime_constraints": {
"docker_image": "bcosc/all-tools",
"min_ram_mb_per_node": 5000
},
I'm using this image (bcosc/all-tools): bcosc@bcosc.qr1hi:~/keep/by_id/3a8f8eee0a3385114f35667ba73112f2+5627$ docker load <37f2378123e6a82b3ef557ed26d337a2f01611b6816aecafe43c7c991cd0304b.tar
I'm trying to install the packages form the arv-base dockerfile (postgresql-server-dev-9.3 instead of 9.1 since 9.1 could not be found):
root@81c0cf71afd8:/# apt-get install -qy apt-utils git curl libcurl3 libcurl3-gnutls libcurl4-openssl-dev locales postgresql-server-dev-9.3 python-arvados-python-client python-google-api-python-client python-oauth2client python-pyasn1=0.1.7 python-pyasn1-modules=0.0.5
Reading package lists...
Building dependency tree...
Reading state information...
git is already the newest version.
locales is already the newest version.
apt-utils is already the newest version.
curl is already the newest version.
libcurl3 is already the newest version.
libcurl3 set to manually installed.
libcurl3-gnutls is already the newest version.
libcurl3-gnutls set to manually installed.
libcurl4-openssl-dev is already the newest version.
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
python-arvados-python-client : Depends: python-pycurl (>= 7.19.5.1) but it is not going to be installed
Depends: python-pycurl (< 7.21.5) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
Updated by Bryan Cosca over 8 years ago
The arvados-python-client is very recent here:
root@dd46aa81f551:/# pip freeze HTSeq==0.6.1p1 Pillow==3.2.0 argparse==1.2.1 arvados-fuse==0.1.20160426220707 arvados-python-client==0.1.20160517202250 biopython==1.66 ciso8601==1.0.1 colorama==0.3.7 cutadapt==1.9.1 docutils==0.12 google-api-python-client==1.5.0 html5lib==0.9999999 httplib2==0.9.2 llfuse==0.41.1 lockfile==0.12.2 numpy==1.11.0 oauth2client==2.0.2 pyasn1==0.1.9 pyasn1-modules==0.0.8 pycurl==7.43.0 pyparsing==2.1.4 pysam==0.9.0 python-daemon==2.1.1 python-dateutil==2.5.3 python-gflags==3.0.5 pytz==2016.4 requests==2.10.0 rsa==3.4.2 simplejson==3.8.2 six==1.10.0 uritemplate==0.6 urllib3==1.15.1 virtualenv==15.0.1 ws4py==0.3.4 wsgiref==0.1.2
Updated by Brett Smith over 8 years ago
Bryan Cosca wrote:
The arvados-python-client is very recent here:
But your version of pycurl is too recent. The Python SDK needs a version of pycurl in between 7.19.5.1 and 7.21.4.
How did you install the Python SDK? Supported installation methods should've detected and dealt with this, unless maybe pycurl got upgraded separately after installing the SDK.
Updated by Bryan Cosca over 8 years ago
I used this to upgrade all my packages: http://stackoverflow.com/questions/2720014/upgrading-all-packages-with-pip. So pycurl probably got upgraded separately after installing the sdk. In any case, I re-installed arvados-python-client and the pycurl version is within the parameters now. (Lesson learned, don't upgrade everything at once)
Updated by Bryan Cosca over 8 years ago
Ran into this:
2016-05-20_19:30:28 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr Using Arvados SDK version 0.1.20160517202250 2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr run-command: bash -c (sed 's/chr//g' /keep/87673c2b7662affafc8657ca1ebf7c8c+98212/PATHOMAP_P00553.vcf) > PATHOMAP_P00553.fixit.vcf 2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr run-command: bash completed with exit code 0 (success) 2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr run-command: the following output files will be saved to keep: 2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr run-command: 676049 ./PATHOMAP_P00553.fixit.vcf 2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr run-command: start writing output to keep 2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr upload wrote 676049 total 676049 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr Traceback (most recent call last): 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr File "/tmp/crunch-job/src/crunch_scripts/run-command", line 433, in <module> 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr outcollection = vwd.checkin(subst.do_substitution(taskp, taskp["task.vwd"]), outdir, merge=True).manifest_text() 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr File "/tmp/crunch-job/src/crunch_scripts/crunchutil/vwd.py", line 50, in checkin 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr cr2 = arvados.CollectionReader(uuid) 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 1623, in __init__ 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr super(CollectionReader, self).__init__(manifest_locator_or_text, *args, **kwargs) 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 1191, in __init__ 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr self._populate() 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 1313, in _populate 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr error_via_keep)) 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr arvados.errors.NotFoundError: Failed to retrieve collection '64d61475e485a88c210fc00dc2d6b3bb+77' from either API server (<HttpError 404 when requesting https://qr1hi.arvadosapi.com/arvados/v1/collections/64d61475e485a88c210fc00dc2d6b3bb%2B77?alt=json returned "Path not found">) or Keep (64d61475e485a88c210fc00dc2d6b3bb+77 not found: http://keep14.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep11.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep23.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep20.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep18.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep22.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep12.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep17.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep25.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep10.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep13.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep16.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep27.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep21.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep26.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep15.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep19.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ; http://keep24.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ). 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 child 21184 on compute8.1 exit 1 success= 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output. 2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 failure (#1, permanent) after 2 seconds
Updated by Brett Smith over 8 years ago
The last error happened because your Crunch script is out of sync with the SDK.
If you can update the pipeline template so that any component that uses run-command uses the arvados repository at a commit that matches the SDK you installed (probably c980683a243903babe9cc09cabc71e1c6229fef1), that will probably work best in the long term.
If that's not possible for any reason, your next best bet is to update the Crunch script and crunchutil in bryan.coscaancestrymapper to match what's currently in the arvados repository.
Updated by Bryan Cosca over 8 years ago
- Target version changed from 2016-05-25 sprint to 2016-06-08 sprint
Updated by Sarah Guthrie over 8 years ago
- Description updated (diff)
- Story points set to 0.5
Updated by Bryan Cosca over 8 years ago
Finishes here: https://workbench.qr1hi.arvadosapi.com/pipeline_instances/qr1hi-d1hrv-5nbuuy223d2gkxc!
Still need to copy logs and outputs to a project and make sure its runnable on a new account.
Updated by Bryan Cosca over 8 years ago
- Status changed from In Progress to Resolved
created a new account bcosctest1 and ran it successfully https://workbench.qr1hi.arvadosapi.com/pipeline_instances/qr1hi-d1hrv-58b2a65p7rn0mvw
Updated by Brett Smith over 8 years ago
Bryan, last week there was a thread on support@ where someone asked about this and I let them know we were working on it. Can you please follow up with them to let them know they can try again, and how to go about that? Thanks in advance.