Bug #5744

PermissionDenied error writing job output of pipeline_instance qr1hi-d1hrv-4zczpddlf3m9uh4

Added by Radhika Chippada almost 5 years ago. Updated almost 5 years ago.

Status:
Closed
Priority:
Normal
Assigned To:
-
Category:
-
Target version:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Story points:
-

Description

This pipeline instance https://cloud.curoverse.com/pipeline_instances/qr1hi-d1hrv-4zczpddlf3m9uh4 failed in the end while writing the output.

According to the logs, the error is a Permission Denied error.

Looking at the API logs, I see the following:
Missing signature on locator e2c9d041e8f3fa360d3efe0af631ca16+67108864

I attached the complete log for this CollectionController#create request found the API server log.

PermissionDenied error log.rtf (7.66 KB) PermissionDenied error log.rtf api server log for this api request Radhika Chippada, 04/15/2015 08:14 PM

Related issues

Is duplicate of Arvados - Bug #3261: [Keep] arv-put fails if permission system is enabledResolved07/31/2014

History

#1 Updated by Radhika Chippada almost 5 years ago

#2 Updated by Bryan Cosca almost 5 years ago

Some notes, this has run succesfully before here: qr1hi-d1hrv-dl3bjgemakkt5aw

Here are some failed instances:

qr1hi-d1hrv-90r2fwvith54p9u
qr1hi-d1hrv-4zczpddlf3m9uh4
qr1hi-d1hrv-y5xjsqr4b32j50h

These users are given read access to the project. They probably tried to run it in their home project but I dont know how to check that.

#3 Updated by Brett Smith almost 5 years ago

  • Status changed from New to Closed
  • Target version deleted (Bug Triage)

This bug is effectively a duplicate of #3261. The Arvados SDK in the Docker image does not retain Keep permission tokens in manifests, and will be rejected by modern API server deployments which require this. Very specifically, they're missing 3bd1f8dad. See #3261-10 note 10, which specifically mentions, "Unfortunately this means docker images with affected versions of the Python SDK can't be used when the permission system is turned up."

qr1hi-d1hrv-dl3bjgemakkt5aw worked because it reused a job from September 2014, before we began enforcing this rule on manifests. The failing pipelines created new jobs and got bit by the bug.

The fix is to either upgrade the Docker image with a newer Arvados SDK version, or update the pipeline to specify a version that at least includes 3bd1f8dad.

Also available in: Atom PDF