Project

General

Profile

Actions

Bug #15814

open

Running a workflow from WB2 exposes secret inputs

Added by Bryan Cosca over 4 years ago. Updated 2 days ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Workbench2
Story points:
-

Description

In the container.json log of e51c5-xvhdp-ctxc9vlsbq7ae3x you can see:

                "api_key_per_sample": {
                    "$include": "/secrets/s0" 
                }

In the container.json log of e51c5-xvhdp-mkzv6lz6mnphv7b, you can see the api_key_per_sample in plaintext.

For reference, the way secrets are handled in arvados-cwl-runner:

  1. the submitter process takes the secret string and adds a "text" type mount at /secrets/s0 (s1, s2, etc) to the container request
  2. In the input object, the parameter is replaced with "$include": "/secrets/s0"
  3. The workflow runner process (inside the container) loads the input object and processes the $input directive, which reads /secrets/s0 and replaces it with the contents of the file
  4. The workflow runner internally swaps the secret for a placeholder to avoid printing it in logging (including debug logging)
  5. The command line tool uses InitialWorkDir to define the credential files
  6. It observes that the file contains the placeholder for the secret
  7. The file is moved to secret_mounts and the placeholder is replaced by the real secret
  8. secret_mounts are hidden from all API responses except when crunch-run requests the "self" container. Secrets are wiped from the database when the container is finished

Implementation

The part that workbench 2 needs to handle is:

  1. Recognizing which inputs are secrets (requires looking for cwltool:Secrets in the workflow's hints or requirements sections).
  2. Obscuring the secret with a "password" type text box
  3. When constructing the container request, moving secrets into the "secret_mounts" part, and replacing them in the input object with the $include reference.

Subtasks 1 (1 open0 closed)

Task #18709: ReviewNewPeter AmstutzActions

Related issues

Related to Arvados - Bug #20977: a-c-r crashes with "Secret store only accepts strings" if you try to register a workflow with secretsNewActions
Actions #1

Updated by Tom Morris over 4 years ago

  • Target version set to 2020-01-02 Sprint
Actions #2

Updated by Tom Morris over 4 years ago

  • Target version changed from 2020-01-02 Sprint to 2020-01-15 Sprint
Actions #3

Updated by Peter Amstutz over 4 years ago

The broader problem is that specifying secrets requires some extra work when creating the container request, and this code is essentially duplicated between arvados-cwl-runner and workbench. As a result, a-c-r knows how to do it, and workbench(1|2) doesn't. This almost certainly isn't the only example of inconsistent behavior between them.

I recommend exploring the option of submitting the workflow using a high level API, specifically the GA4GH Workflow Execution Service (WES). The implementation of the high level API can invoke arvados-cwl-runner which already has the logic to construct the low level container request. Submitted for consideration: #15918 #15917

Actions #4

Updated by Peter Amstutz over 4 years ago

  • Target version changed from 2020-01-15 Sprint to Arvados Future Sprints
Actions #5

Updated by Peter Amstutz about 4 years ago

  • Release set to 20
  • Target version deleted (Arvados Future Sprints)
Actions #6

Updated by Peter Amstutz about 2 years ago

  • Release changed from 20 to 46
  • Target version set to 2022-02-16 sprint
Actions #7

Updated by Peter Amstutz about 2 years ago

  • Assigned To set to Peter Amstutz
Actions #8

Updated by Peter Amstutz about 2 years ago

  • Subject changed from Running a workflow from WB1 & WB2 exposes secret inputs to Running a workflow from WB2 exposes secret inputs
Actions #9

Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)
Actions #10

Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)
Actions #11

Updated by Peter Amstutz about 2 years ago

  • Target version changed from 2022-02-16 sprint to 2022-03-02 sprint
Actions #12

Updated by Peter Amstutz about 2 years ago

  • Target version changed from 2022-03-02 sprint to 2022-03-16 sprint
Actions #13

Updated by Peter Amstutz about 2 years ago

  • Description updated (diff)
Actions #14

Updated by Peter Amstutz about 2 years ago

  • Project changed from 35 to Arvados
Actions #15

Updated by Peter Amstutz about 2 years ago

  • Assigned To changed from Peter Amstutz to Daniel Kutyła
Actions #16

Updated by Peter Amstutz about 2 years ago

  • Target version changed from 2022-03-16 sprint to 2022-04-13 Sprint
Actions #17

Updated by Peter Amstutz about 2 years ago

  • Release changed from 46 to 51
Actions #18

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-04-13 Sprint to 2022-04-27 Sprint
Actions #19

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-04-27 Sprint to 2022-05-11 sprint
Actions #20

Updated by Peter Amstutz almost 2 years ago

  • Release deleted (51)
Actions #21

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-05-11 sprint to 2022-05-25 sprint
Actions #22

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-05-25 sprint to 2022-06-08 sprint
Actions #23

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-06-08 sprint to 2022-06-22 Sprint
Actions #24

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-06-22 Sprint to 2022-07-06
Actions #25

Updated by Peter Amstutz almost 2 years ago

  • Target version changed from 2022-07-06 to 2022-07-20
Actions #26

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-07-20 to 2022-08-03 Sprint
Actions #27

Updated by Peter Amstutz over 1 year ago

  • Category set to Workbench2
Actions #28

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-08-03 Sprint to 2022-08-17 sprint
Actions #29

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-08-17 sprint to 2022-08-31 sprint
Actions #30

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-08-31 sprint to 2022-09-28 sprint
Actions #31

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-09-28 sprint to 2022-10-12 sprint
Actions #32

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-10-12 sprint to 2022-10-26 sprint
Actions #33

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-10-26 sprint to 2022-11-09 sprint
Actions #34

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-11-09 sprint to 2022-11-23 sprint
Actions #35

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-11-23 sprint to 2022-12-21 Sprint
Actions #36

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2022-12-21 Sprint to 2023-01-18 sprint
Actions #37

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2023-01-18 sprint to 2023-02-01 sprint
Actions #38

Updated by Peter Amstutz over 1 year ago

  • Target version changed from 2023-02-01 sprint to 2023-02-15 sprint
Actions #39

Updated by Peter Amstutz about 1 year ago

  • Target version changed from 2023-02-15 sprint to Future
Actions #40

Updated by Peter Amstutz 6 months ago

  • Assigned To deleted (Daniel Kutyła)
Actions #42

Updated by Brett Smith 6 months ago

  • Related to Bug #20977: a-c-r crashes with "Secret store only accepts strings" if you try to register a workflow with secrets added
Actions #43

Updated by Peter Amstutz 4 months ago

  • Target version changed from Future to Development 2024-01-17 sprint
Actions #44

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-01-17 sprint to Development 2024-02-28 sprint
Actions #45

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-02-28 sprint to Development 2024-01-31 sprint
Actions #46

Updated by Peter Amstutz 4 months ago

  • Target version changed from Development 2024-01-31 sprint to Development 2024-02-28 sprint
Actions #47

Updated by Peter Amstutz about 2 months ago

  • Target version changed from Development 2024-02-28 sprint to Development 2024-03-13 sprint
Actions #48

Updated by Peter Amstutz about 1 month ago

  • Target version changed from Development 2024-03-13 sprint to Development 2024-03-27 sprint
Actions #49

Updated by Peter Amstutz 27 days ago

  • Target version changed from Development 2024-03-27 sprint to Development 2024-04-10 sprint
Actions #50

Updated by Peter Amstutz 27 days ago

  • Target version changed from Development 2024-04-10 sprint to Development 2024-04-24 sprint
Actions

Also available in: Atom PDF