Project

General

Profile

Actions

Idea #5914

open

[DRAFT] Provide one clear way for users to get data from an external source into Arvados

Added by Brett Smith over 9 years ago. Updated 8 months ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
Documentation
Target version:
Start date:
Due date:
Story points:
-
Release:
Release relationship:
Auto

Description

Background

Users frequently want to work with data available from public sites. It's important that it be easy for them to get it into Arvados so they can start processing on it as quickly as possible.

Proposed solution

A page of documentation, a step-by-step walkthrough describing the best way to do this.

  • How to get the data with wget, including all the flags you want to use to download the data reliably over an unreliable link, with as much metadata as possible (like Last-Modified dates)
  • How to get the data into Arvados - probably with the writable FUSE mount
  • How to get updates - this can be with wget if all the right metadata is available, otherwise… this is where the story gets a little fuzzy.
Actions #1

Updated by Brett Smith over 9 years ago

  • Subject changed from [DRAFT] Proivde one clear way for users to get data from an external source into Arvados to [DRAFT] Provide one clear way for users to get data from an external source into Arvados
Actions #2

Updated by Ward Vandewege over 3 years ago

  • Target version deleted (Arvados Future Sprints)
Actions #3

Updated by Peter Amstutz over 1 year ago

  • Release set to 60
Actions #4

Updated by Peter Amstutz 8 months ago

  • Target version set to Future
Actions

Also available in: Atom PDF