Bug #3261

[Keep] arv-put fails if permission system is enabled

Added by Tom Clegg about 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Keep
Target version:
Start date:
07/31/2014
Due date:
% Done:

100%

Estimated time:
(Total: 14.00 h)
Story points:
1.0

Description

In production, when Keep and the API server are configured to enforce permissions: after storing a collection in Keep, arv-put attempts to write the new collection to the API server but it fails a permission signature validation check.


Subtasks

Task #3274: Confirm bug still exists by reproducing somewhereResolvedWard Vandewege

Task #3429: Review 3261-keep-docker-configResolvedWard Vandewege

Task #3626: Update docker images, esp. base and java-bwa-samtoolsResolvedWard Vandewege

Task #3707: Ensure crunch-job task output collation can survive with all permission features enabledResolved


Related issues

Has duplicate Arvados - Bug #5744: PermissionDenied error writing job output of pipeline_instance qr1hi-d1hrv-4zczpddlf3m9uh4 Closed

Associated revisions

Revision 5f6af45e (diff)
Added by Tim Pierce about 5 years ago

3261: fixes for Docker build process

  • New tool list_generated_files avoids difficult quoting/escaping
    problems in Makefile
  • keep/Dockerfile fixes permissions on /usr/local/bin/run-keep

Refs #3261.

Revision 88442524
Added by Ward Vandewege about 5 years ago

Merge branch '3261-keep-docker-config'

refs #3261

Revision 3bd1f8da (diff)
Added by Tom Clegg about 5 years ago

Do not throw away permission hints when writing data. refs #3261

Revision accaf5ae (diff)
Added by Tom Clegg about 5 years ago

Do not try to predict uuid of new collection. refs #3261 refs #3504

History

#1 Updated by Tim Pierce about 5 years ago

  • Assigned To set to Tim Pierce

#2 Updated by Tim Pierce about 5 years ago

  • Description updated (diff)

#3 Updated by Tim Pierce about 5 years ago

Reviewing tomclegg's 3261-keep-docker-config branch at 7305149:

  • docker/api/omniauth.rb.in
    • I don't think that SSO_HOSTNAME.ARVADOS_DOMAIN is going to resolve to the local SSO container, which is what ENV['SSO_PORT_443_TCP_ADDR'] did.
  • docker/build_tools/Makefile
    • This line:
          KEEP_GENERATED := $(shell ls -1 $(KEEP_GENERATED_IN) | perl -pe 's:^keep:keep/generated:')
      
    • ... I think also needs s/\.in$//.
  • docker/keep/run-keep.in
    • -z is the wrong test: this looks like it should be
      if [ -s "$pkf" ]
      then
          permisson_key_arg="-permission-key-file=$pkf" 
      else
          permisson_key_arg="" 
      fi
      

#4 Updated by Tom Clegg about 5 years ago

Tim Pierce wrote:

  • docker/api/omniauth.rb.in
    • I don't think that SSO_HOSTNAME.ARVADOS_DOMAIN is going to resolve to the local SSO container, which is what ENV['SSO_PORT_443_TCP_ADDR'] did.
Ah, I wondered where that came from. I've added that as a fallback, and changed the config.yml default to empty, so we have two main options:
  • Leave empty in config.yml = use sso docker container
  • Specify https://auth.curoverse.com in config.yml = use some sso service elsewhere
  • docker/build_tools/Makefile
    • This line:
      [...]
    • ... I think also needs s/\.in$//.

Ah yes. (Now obsoleted by your replacement.)

  • docker/keep/run-keep.in
    • -z is the wrong test: this looks like it should be
      [...]

Yes. Fixed, thanks.

With these and other misc fixes, I've confirmed that #3261 is not so easily reproduced. :/

#5 Updated by Tom Clegg about 5 years ago

After turning on --enforce-permissions, both keep and api server seem to work as expected:

keep

root@compute0:/# curl -H 'Authorization: OAuth2 4mqswa8myo058hxxfyx0r9zt9mm5m91brpviuvaoun4d857ewd' http://keep_server_0.keep.dev.arvados:25107/acbd18db4cc2f85cedef654fccc4a4d8+3+A8dd47f55a2815eb149887938d3416e5fdb6f9813@53ed4abf
foo

root@compute0:/# curl -H 'Authorization: OAuth2 4mqswa8myo058hxxfyx0r9zt9mm5m91brpviuvaoun4d857ewd' http://keep_server_0.keep.dev.arvados:25107/acbd18db4cc2f85cedef654fccc4a4d8+3+A8dd47f55a2815eb149887938d3416e5fdb6fffff@53ed4abf
Forbidden

api

root@compute0:/tmp# arv collection create --collection '{"uuid":"1f4b0bc7583c2a7f9102c395f4ffc5e3+45","manifest_text":". acbd18db4cc2f85cedef654fccc4a4d8+3+Afa79eed615b0cadac0a86dcd99c155c49d654c43@53ed418c 0:3:foo\n"}'
# => success

root@compute0:/tmp# arv collection create --collection '{"uuid":"1f4b0bc7583c2a7f9102c395f4ffc5e3+45","manifest_text":". acbd18db4cc2f85cedef654fccc4a4d8+3+A1dcc4d36024c3c4166088ba32686ffb428c33fc2@53ed4163 0:3:foo\n"}'
Error: #<ArvadosModel::PermissionDeniedError: ArvadosModel::PermissionDeniedError>

#6 Updated by Tom Clegg about 5 years ago

Turning on permissions on 4xphq also seems fine:

4xphq:~# echo -n bar > bar
4xphq:~# arv-put bar
arv-put: No --project-uuid or --name specified.  This data will be cached
in Keep.  You will need to find this upload by its locator(s) later.
0M / 0M 100.0% 
fa7aeb5140e2848d39b416daeef4ffc5+45

4xphq:~# arv-get fa7aeb5140e2848d39b416daeef4ffc5+45
. 37b51d194a7513e45b56f6524f2d51f2+3+Af291968a5e126295afeef53e71e4537a2e6d7fa3@53ed4fd8 0:3:bar

4xphq:~# arv collection create --collection '{"uuid":"fa7aeb5140e2848d39b416daeef4ffc5+45","manifest_text":". 37b51d194a7513e45b56f6524f2d51f2+3+Af291968a5e126295afeef53e71e4537a2e6d7fa3@53ed4fd8 0:3:bar\n"}'
{
 "href":"/collections/fa7aeb5140e2848d39b416daeef4ffc5+45",
 "kind":"arvados#collection",
 ...

4xphq:~# arv collection create --collection '{"uuid":"fa7aeb5140e2848d39b416daeef4ffc5+45","manifest_text":". 37b51d194a7513e45b56f6524f2d51f2+3+Af291968a5e126295afeef53e71e4537a2e6dffff@53ed4fd8 0:3:bar\n"}'
Error: #<ArvadosModel::PermissionDeniedError: ArvadosModel::PermissionDeniedError>

#7 Updated by Tom Clegg about 5 years ago

Meanwhile, more docker patches in 3261-keep-docker-config @ 5ce0e0f

#8 Updated by Tom Clegg about 5 years ago

  • Category set to Keep
  • Status changed from New to In Progress
  • Assigned To changed from Tim Pierce to Ward Vandewege

#9 Updated by Ward Vandewege about 5 years ago

  • Target version changed from 2014-08-06 Sprint to 2014-08-27 Sprint

#10 Updated by Tom Clegg about 5 years ago

  • Bug: CollectionWriter.finish() (as used by well-behaved crunch scripts, but not crunch-job) threw away its permission hints before showing manifest_text to anyone, so of course arvados.v1.collections.create failed.
  • Fix: Don't do that. 3bd1f8da
  • Unfortunately this means docker images with affected versions of the Python SDK can't be used when the permission system is turned up.

#11 Updated by Tom Clegg almost 5 years ago

  • Target version changed from 2014-08-27 Sprint to 2014-09-17 sprint

#12 Updated by Tom Clegg almost 5 years ago

  • Target version changed from 2014-09-17 sprint to Arvados Future Sprints

#13 Updated by Tom Clegg almost 5 years ago

  • Target version changed from Arvados Future Sprints to 2014-10-08 sprint

#14 Updated by Ward Vandewege almost 5 years ago

  • Status changed from In Progress to Resolved

#15 Updated by Ward Vandewege almost 5 years ago

  • Status changed from Resolved to In Progress

#16 Updated by Ward Vandewege almost 5 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF