Feature #8688

Accept authenticated API calls from data providers to add datasets to a public profile

Added by Ward Vandewege almost 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
Third party integration
Start date:
03/24/2016
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
2.0

Description

The API call will have these arguments:

a) a unique identifier for the external data provider

b) the 3rd party activity identifier for the pgp participant which was sent to the data provider when the participant signed up for the activity

c) a payload which indicates what file(s) to import. Details of the format of that payload are TBD, but at a minimum it should probably be a collection of URLs, ideally associated with a file type for each individual file to download. The URLs should include either a manifest file with MD5 digests (like http://cdimage.debian.org/debian-cd/8.3.0/amd64/iso-cd/MD5SUMS), or an *.md5 file for each data file. The size of each file should also be included.

Acceptance criteria

When a researcher or admin shows/edits a third party study, the unique identifier for the study is shown. (This is passed as api_key.)

Given a POST /third_party/add_dataset request:
  1. Ensure api_key corresponds to an approved third-party study
  2. Ensure participant_code param corresponds to a participant who is participating in the study (otherwise HTTP 4xx)
  3. Ensure data_sources_tsv param is a TSV document, where each line has a URI, size, and hex md5 digest of the corresponding file data
  4. Respond HTTP 4xx if any parameters are invalid or there is an error submitting the download job; otherwise 200

Subtasks

Task #8792: Review 8688-pull-data-apiResolvedWard Vandewege


Related issues

Related to Tapestry - Feature #8689: Trigger an arvados job to retrieve the data and import it into KeepIn Progress03/14/2016

Related to Tapestry - Feature #8707: Arvados job: download data from remote site into KeepIn Progress03/15/2016

Associated revisions

Revision c15f3953
Added by Tom Clegg almost 4 years ago

Merge branch '8688-pull-data-api'

refs #8688

Revision 8b161241 (diff)
Added by Tom Clegg almost 4 years ago

8688: Rename param from participant_code to participant_id
to match "clickthrough to participate" link.

refs #8688

History

#1 Updated by Ward Vandewege almost 4 years ago

  • Description updated (diff)

#2 Updated by Ward Vandewege almost 4 years ago

  • Description updated (diff)

#3 Updated by Tom Clegg almost 4 years ago

  • Description updated (diff)

#4 Updated by Tom Clegg almost 4 years ago

  • Description updated (diff)

#5 Updated by Tom Clegg almost 4 years ago

  • Description updated (diff)

#6 Updated by Abram Connelly almost 4 years ago

  • Description updated (diff)

#7 Updated by Tom Clegg almost 4 years ago

  • Story points set to 2.0

#8 Updated by Tom Clegg almost 4 years ago

  • Assigned To set to Tom Clegg
  • Category set to Third party integration

#9 Updated by Tom Clegg almost 4 years ago

  • Description updated (diff)

#10 Updated by Tom Clegg almost 4 years ago

  • Status changed from New to In Progress

#11 Updated by Tom Clegg almost 4 years ago

  • Description updated (diff)
  • Status changed from In Progress to New

#12 Updated by Tom Clegg almost 4 years ago

  • Status changed from New to In Progress

#13 Updated by Tom Clegg almost 4 years ago

  • Description updated (diff)
Branch 8688-pull-data-api @ 4969df6
  • Accepts valid API calls: POST /third_party/add_dataset with params api_key, data_sources_tsv, participant_code
  • Rejects invalid API calls with 4xx and a JSON response body
  • Tries to submit an Arvados pipeline using arv-run-pipeline-instance (arvados-cli must be installed, but this dependency can't be in Gemfile because it needs Ruby 1.9.3) (this is part of #8689)
  • Introduces a new config knob ruby19_shell_args (default "rvm-exec 1.9.3")

#14 Updated by Ward Vandewege almost 4 years ago

I looked at 8688-pull-data-api, and I think it's OK to merge. When merged I can deploy to my-dev and we can test functionality. Thanks for adding a journal table!

#15 Updated by Tom Clegg almost 4 years ago

  • Status changed from In Progress to Resolved

Also available in: Atom PDF