Project

General

Profile

Actions
Copy link

Bug #9233

closed

Fix Ancestry Mapper pipeline

Added by Bryan Cosca almost 8 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Bryan Cosca
Category:
-
Target version:
2016-06-08 sprint
Story points:
0.5

Description

The Ancestry Mapper Pipeline no longer completes:

Currently, the pipeline fails on SNP-Annotation-justmap (missing an input). All the failures so far have been due to outdated run-command problems, so this story is not expected to take a long time, even though the work is not completely determined.

Latest run of pipeline: https://workbench.qr1hi.arvadosapi.com/pipeline_instances/qr1hi-d1hrv-q75kebcrxhex370

This story is considered complete when it completes successfully on its default inputs and the updated pipeline is in the Public Project.

Actions
Copy link
 #1

Updated by Bryan Cosca almost 8 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by Bryan Cosca almost 8 years ago

  • Assigned To set to Bryan Cosca
Actions
Copy link
 #3

Updated by Bryan Cosca almost 8 years ago

Updated a bunch of packages on the docker image and ran into this:

2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr   File "/tmp/crunch-job/src/crunch_scripts/run-command", line 42, in <module>
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr     t = arvados.current_task().tmpdir
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados/__init__.py", line 65, in current_task
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr     task = api('v1').job_tasks().get(uuid=os.environ['TASK_UUID']).execute()
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr   File "/usr/local/lib/python2.7/dist-packages/oauth2client/util.py", line 135, in positional_wrapper
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr     return wrapped(*args, **kwargs)
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr   File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 755, in execute
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr     method=str(self.method), body=self.body, headers=self.headers)
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr   File "/usr/local/lib/python2.7/dist-packages/googleapiclient/http.py", line 93, in _retry_request
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr     resp, content = http.request(uri, method, *args, **kwargs)
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr TypeError: _intercept_http_request() takes exactly 2 arguments (5 given)
2016-05-18_21:08:46 qr1hi-8i9sb-mo15okvl037dknp 30402 0 stderr srun: error: compute4: task 0: Exited with exit code 1
Actions
Copy link
 #4

Updated by Bryan Cosca almost 8 years ago

I'm not specifying arvados_sdk_version in the template, and am running into this error still: instance: qr1hi-d1hrv-n3f1ow28sdj58qr job: qr1hi-8i9sb-mo15okvl037dknp

    "RemoveIndels-fixit": {
      "script": "run-command",
      "script_version": "79509812aefb7c32cb7755adc77a716b8491dec8",
      "repository": "bryan.coscamasonancestry",
      "script_parameters": {
        "command": {
          "value": [
            "bash",
            "-c",
            "(sed 's/chr//g' $(glob $(dir $(input))/*.vcf)) > $(basename $(glob $(dir $(input))/*.vcf)).fixit.vcf" 
          ]
        },
        "input": {
          "required": true,
          "dataclass": "Collection",
          "title": "Input VCF file + 00-all.vcf + 1KG VCFs",
          "default": "87673c2b7662affafc8657ca1ebf7c8c+98212",
          "value": "87673c2b7662affafc8657ca1ebf7c8c+98212" 
        },
        "task.vwd": {
          "value": "$(input)" 
        }
      },
      "runtime_constraints": {
        "docker_image": "bcosc/all-tools",
        "min_ram_mb_per_node": 5000
      },

I'm using this image (bcosc/all-tools): :~/keep/by_id/3a8f8eee0a3385114f35667ba73112f2+5627$ docker load <37f2378123e6a82b3ef557ed26d337a2f01611b6816aecafe43c7c991cd0304b.tar

I'm trying to install the packages form the arv-base dockerfile (postgresql-server-dev-9.3 instead of 9.1 since 9.1 could not be found):

root@81c0cf71afd8:/# apt-get install -qy apt-utils git curl libcurl3 libcurl3-gnutls libcurl4-openssl-dev locales postgresql-server-dev-9.3 python-arvados-python-client python-google-api-python-client python-oauth2client python-pyasn1=0.1.7 python-pyasn1-modules=0.0.5
Reading package lists...
Building dependency tree...
Reading state information...
git is already the newest version.
locales is already the newest version.
apt-utils is already the newest version.
curl is already the newest version.
libcurl3 is already the newest version.
libcurl3 set to manually installed.
libcurl3-gnutls is already the newest version.
libcurl3-gnutls set to manually installed.
libcurl4-openssl-dev is already the newest version.
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 python-arvados-python-client : Depends: python-pycurl (>= 7.19.5.1) but it is not going to be installed
                                Depends: python-pycurl (< 7.21.5) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
Actions
Copy link
 #5

Updated by Bryan Cosca almost 8 years ago

The arvados-python-client is very recent here:

root@dd46aa81f551:/# pip freeze 
HTSeq==0.6.1p1
Pillow==3.2.0
argparse==1.2.1
arvados-fuse==0.1.20160426220707
arvados-python-client==0.1.20160517202250
biopython==1.66
ciso8601==1.0.1
colorama==0.3.7
cutadapt==1.9.1
docutils==0.12
google-api-python-client==1.5.0
html5lib==0.9999999
httplib2==0.9.2
llfuse==0.41.1
lockfile==0.12.2
numpy==1.11.0
oauth2client==2.0.2
pyasn1==0.1.9
pyasn1-modules==0.0.8
pycurl==7.43.0
pyparsing==2.1.4
pysam==0.9.0
python-daemon==2.1.1
python-dateutil==2.5.3
python-gflags==3.0.5
pytz==2016.4
requests==2.10.0
rsa==3.4.2
simplejson==3.8.2
six==1.10.0
uritemplate==0.6
urllib3==1.15.1
virtualenv==15.0.1
ws4py==0.3.4
wsgiref==0.1.2
Actions
Copy link
 #6

Updated by Brett Smith almost 8 years ago

Bryan Cosca wrote:

The arvados-python-client is very recent here:

But your version of pycurl is too recent. The Python SDK needs a version of pycurl in between 7.19.5.1 and 7.21.4.

How did you install the Python SDK? Supported installation methods should've detected and dealt with this, unless maybe pycurl got upgraded separately after installing the SDK.

Actions
Copy link
 #7

Updated by Bryan Cosca almost 8 years ago

I used this to upgrade all my packages: http://stackoverflow.com/questions/2720014/upgrading-all-packages-with-pip. So pycurl probably got upgraded separately after installing the sdk. In any case, I re-installed arvados-python-client and the pycurl version is within the parameters now. (Lesson learned, don't upgrade everything at once)

Actions
Copy link
 #8

Updated by Bryan Cosca almost 8 years ago

Ran into this:

2016-05-20_19:30:28 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr Using Arvados SDK version 0.1.20160517202250
2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr run-command: bash -c (sed 's/chr//g' /keep/87673c2b7662affafc8657ca1ebf7c8c+98212/PATHOMAP_P00553.vcf) > PATHOMAP_P00553.fixit.vcf
2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr run-command: bash completed with exit code 0 (success)
2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr run-command: the following output files will be saved to keep:
2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr run-command:       676049 ./PATHOMAP_P00553.fixit.vcf
2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr run-command: start writing output to keep
2016-05-20_19:30:29 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr upload wrote 676049 total 676049
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr Traceback (most recent call last):
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr   File "/tmp/crunch-job/src/crunch_scripts/run-command", line 433, in <module>
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr     outcollection = vwd.checkin(subst.do_substitution(taskp, taskp["task.vwd"]), outdir, merge=True).manifest_text()
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr   File "/tmp/crunch-job/src/crunch_scripts/crunchutil/vwd.py", line 50, in checkin
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr     cr2 = arvados.CollectionReader(uuid)
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 1623, in __init__
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr     super(CollectionReader, self).__init__(manifest_locator_or_text, *args, **kwargs)
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 1191, in __init__
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr     self._populate()
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr   File "/usr/local/lib/python2.7/dist-packages/arvados/collection.py", line 1313, in _populate
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr     error_via_keep))
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr arvados.errors.NotFoundError: Failed to retrieve collection '64d61475e485a88c210fc00dc2d6b3bb+77' from either API server (<HttpError 404 when requesting https://qr1hi.arvadosapi.com/arvados/v1/collections/64d61475e485a88c210fc00dc2d6b3bb%2B77?alt=json returned "Path not found">) or Keep (64d61475e485a88c210fc00dc2d6b3bb+77 not found:  http://keep14.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep11.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep23.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep20.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep18.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep22.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep12.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep17.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep25.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep10.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep13.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep16.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep27.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep21.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep26.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep15.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep19.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ;  http://keep24.qr1hi.arvadosapi.com:25107/ responded with 403 HTTP/1.1 403 Forbidden
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 stderr ).
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 child 21184 on compute8.1 exit 1 success=
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 ERROR: Task process exited 1, but never updated its task record to indicate success and record its output.
2016-05-20_19:30:30 qr1hi-8i9sb-3vlaqwm243dxhuv 21056 0 failure (#1, permanent) after 2 seconds
Actions
Copy link
 #9

Updated by Brett Smith almost 8 years ago

The last error happened because your Crunch script is out of sync with the SDK.

If you can update the pipeline template so that any component that uses run-command uses the arvados repository at a commit that matches the SDK you installed (probably c980683a243903babe9cc09cabc71e1c6229fef1), that will probably work best in the long term.

If that's not possible for any reason, your next best bet is to update the Crunch script and crunchutil in bryan.coscaancestrymapper to match what's currently in the arvados repository.

Actions
Copy link
 #10

Updated by Bryan Cosca almost 8 years ago

  • Target version changed from 2016-05-25 sprint to 2016-06-08 sprint
Actions
Copy link
 #11

Updated by Sarah Guthrie almost 8 years ago

  • Description updated (diff)
  • Story points set to 0.5
Actions
Copy link
 #12

Updated by Sarah Guthrie almost 8 years ago

  • Description updated (diff)
Actions
Copy link
 #13

Updated by Bryan Cosca almost 8 years ago

Finishes here: https://workbench.qr1hi.arvadosapi.com/pipeline_instances/qr1hi-d1hrv-5nbuuy223d2gkxc!

Still need to copy logs and outputs to a project and make sure its runnable on a new account.

Actions
Copy link
 #14

Updated by Bryan Cosca almost 8 years ago

  • Status changed from In Progress to Resolved

created a new account bcosctest1 and ran it successfully https://workbench.qr1hi.arvadosapi.com/pipeline_instances/qr1hi-d1hrv-58b2a65p7rn0mvw

Actions
Copy link
 #15

Updated by Brett Smith almost 8 years ago

Bryan, last week there was a thread on support@ where someone asked about this and I let them know we were working on it. Can you please follow up with them to let them know they can try again, and how to go about that? Thanks in advance.

Actions
Copy link

Also available in: Atom PDF