Project

General

Profile

Actions

Bug #15814

open

Running a workflow from WB2 exposes secret inputs

Added by Bryan Cosca about 3 years ago. Updated 3 days ago.

Status:
New
Priority:
Normal
Assigned To:
Category:
Workbench2
Target version:
Start date:
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Story points:
-

Description

In the container.json log of e51c5-xvhdp-ctxc9vlsbq7ae3x you can see:

                "api_key_per_sample": {
                    "$include": "/secrets/s0" 
                }

In the container.json log of e51c5-xvhdp-mkzv6lz6mnphv7b, you can see the api_key_per_sample in plaintext.

For reference, the way secrets are handled in arvados-cwl-runner:

  1. the submitter process takes the secret string and adds a "text" type mount at /secrets/s0 (s1, s2, etc) to the container request
  2. In the input object, the parameter is replaced with "$include": "/secrets/s0"
  3. The workflow runner process (inside the container) loads the input object and processes the $input directive, which reads /secrets/s0 and replaces it with the contents of the file
  4. The workflow runner internally swaps the secret for a placeholder to avoid printing it in logging (including debug logging)
  5. The command line tool uses InitialWorkDir to define the credential files
  6. It observes that the file contains the placeholder for the secret
  7. The file is moved to secret_mounts and the placeholder is replaced by the real secret
  8. secret_mounts are hidden from all API responses except when crunch-run requests the "self" container. Secrets are wiped from the database when the container is finished

Implementation

The part that workbench 2 needs to handle is:

  1. Recognizing which inputs are secrets (requires looking for cwltool:Secrets in the workflow's hints or requirements sections).
  2. Obscuring the secret with a "password" type text box
  3. When constructing the container request, moving secrets into the "secret_mounts" part, and replacing them in the input object with the $include reference.

Subtasks 1 (1 open0 closed)

Task #18709: ReviewNewPeter Amstutz

Actions
Actions #1

Updated by Tom Morris almost 3 years ago

  • Target version set to 2020-01-02 Sprint
Actions #2

Updated by Tom Morris almost 3 years ago

  • Target version changed from 2020-01-02 Sprint to 2020-01-15 Sprint
Actions #3

Updated by Peter Amstutz almost 3 years ago

The broader problem is that specifying secrets requires some extra work when creating the container request, and this code is essentially duplicated between arvados-cwl-runner and workbench. As a result, a-c-r knows how to do it, and workbench(1|2) doesn't. This almost certainly isn't the only example of inconsistent behavior between them.

I recommend exploring the option of submitting the workflow using a high level API, specifically the GA4GH Workflow Execution Service (WES). The implementation of the high level API can invoke arvados-cwl-runner which already has the logic to construct the low level container request. Submitted for consideration: #15918 #15917

Actions #4

Updated by Peter Amstutz almost 3 years ago

  • Target version changed from 2020-01-15 Sprint to Arvados Future Sprints
Actions #5

Updated by Peter Amstutz almost 3 years ago

  • Release set to 20
  • Target version deleted (Arvados Future Sprints)
Actions #6

Updated by Peter Amstutz 10 months ago

  • Release changed from 20 to 46
  • Target version set to 2022-02-16 sprint
Actions #7

Updated by Peter Amstutz 10 months ago

  • Assigned To set to Peter Amstutz
Actions #8

Updated by Peter Amstutz 10 months ago

  • Subject changed from Running a workflow from WB1 & WB2 exposes secret inputs to Running a workflow from WB2 exposes secret inputs
Actions #9

Updated by Peter Amstutz 10 months ago

  • Description updated (diff)
Actions #10

Updated by Peter Amstutz 10 months ago

  • Description updated (diff)
Actions #11

Updated by Peter Amstutz 9 months ago

  • Target version changed from 2022-02-16 sprint to 2022-03-02 sprint
Actions #12

Updated by Peter Amstutz 9 months ago

  • Target version changed from 2022-03-02 sprint to 2022-03-16 sprint
Actions #13

Updated by Peter Amstutz 9 months ago

  • Description updated (diff)
Actions #14

Updated by Peter Amstutz 9 months ago

  • Project changed from 35 to Arvados
Actions #15

Updated by Peter Amstutz 9 months ago

  • Assigned To changed from Peter Amstutz to Daniel Kutyła
Actions #16

Updated by Peter Amstutz 9 months ago

  • Target version changed from 2022-03-16 sprint to 2022-04-13 Sprint
Actions #17

Updated by Peter Amstutz 8 months ago

  • Release changed from 46 to 51
Actions #18

Updated by Peter Amstutz 8 months ago

  • Target version changed from 2022-04-13 Sprint to 2022-04-27 Sprint
Actions #19

Updated by Peter Amstutz 8 months ago

  • Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint
Actions #20

Updated by Peter Amstutz 7 months ago

  • Release deleted (51)
Actions #21

Updated by Peter Amstutz 7 months ago

  • Target version changed from 2022-05-11 sprint to 2022-05-25 sprint
Actions #22

Updated by Peter Amstutz 7 months ago

  • Target version changed from 2022-05-25 sprint to 2022-06-08 sprint
Actions #23

Updated by Peter Amstutz 7 months ago

  • Target version changed from 2022-06-08 sprint to 2022-06-22 Sprint
Actions #24

Updated by Peter Amstutz 7 months ago

  • Target version changed from 2022-06-22 Sprint to 2022-07-06
Actions #25

Updated by Peter Amstutz 5 months ago

  • Target version changed from 2022-07-06 to 2022-07-20
Actions #26

Updated by Peter Amstutz 5 months ago

  • Target version changed from 2022-07-20 to 2022-08-03 Sprint
Actions #27

Updated by Peter Amstutz 5 months ago

  • Category set to Workbench2
Actions #28

Updated by Peter Amstutz 5 months ago

  • Target version changed from 2022-08-03 Sprint to 2022-08-17 sprint
Actions #29

Updated by Peter Amstutz 5 months ago

  • Target version changed from 2022-08-17 sprint to 2022-08-31 sprint
Actions #30

Updated by Peter Amstutz 4 months ago

  • Target version changed from 2022-08-31 sprint to 2022-09-28 sprint
Actions #31

Updated by Peter Amstutz 2 months ago

  • Target version changed from 2022-09-28 sprint to 2022-10-12 sprint
Actions #32

Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2022-10-12 sprint to 2022-10-26 sprint
Actions #33

Updated by Peter Amstutz about 2 months ago

  • Target version changed from 2022-10-26 sprint to 2022-11-09 sprint
Actions #34

Updated by Peter Amstutz 18 days ago

  • Target version changed from 2022-11-09 sprint to 2022-11-23 sprint
Actions #35

Updated by Peter Amstutz 18 days ago

  • Target version changed from 2022-11-23 sprint to 2022-12-21 Sprint
Actions #36

Updated by Peter Amstutz 4 days ago

  • Target version changed from 2022-12-21 Sprint to 2023-01-18 sprint
Actions #37

Updated by Peter Amstutz 3 days ago

  • Target version changed from 2023-01-18 sprint to 2023-02-01 sprint
Actions #38

Updated by Peter Amstutz 3 days ago

  • Target version changed from 2023-02-01 sprint to 2023-02-15 sprint
Actions

Also available in: Atom PDF