Actions
Idea #15960
openComputing on external data
Story points:
-
Release:
Release relationship:
Auto
Description
Right now, the feature of automatic HTTP download in cwl-runner
is effectively fulfilling this function for users (although it copies it into the local keepstore). Users would probably like it if it were expanded to also support copying s3:// URLs.
However, the big idea for this epic is on-demand retrieval from external storage -- we fetch the data from the external system on demand.
This involves:
- Going through each file of an external file system or bucket and hashing 64 MB byte ranges to get a block hash
- Keeping a database that maps the block hash to a URL and byte range
- Constructing a collection using these blocks with file/directory structure matching the external file system
- Creating a special keepstore volume type that uses the block hash database to fetch block contents from the external source on demand as if it were a normal keep service
Related issues
Updated by Peter Amstutz about 4 years ago
- Start date set to 04/01/2020
- Due date set to 06/30/2020
Updated by Peter Amstutz about 4 years ago
- Start date changed from 04/01/2020 to 05/01/2020
- Due date changed from 06/30/2020 to 07/31/2020
Updated by Peter Amstutz about 4 years ago
- Start date changed from 05/01/2020 to 04/01/2020
- Due date changed from 07/31/2020 to 06/30/2020
Updated by Peter Amstutz about 4 years ago
- Start date changed from 04/01/2020 to 05/01/2020
- Due date changed from 06/30/2020 to 07/31/2020
Updated by Peter Amstutz about 4 years ago
- Related to Feature #8570: [Crunch2] Impure access to object store added
Updated by Peter Amstutz about 4 years ago
- Related to Feature #8569: [Crunch2] Impure mount from host fs added
Updated by Peter Amstutz about 4 years ago
- Start date changed from 05/01/2020 to 04/01/2020
Updated by Peter Amstutz about 4 years ago
- Due date changed from 07/31/2020 to 07/01/2020
Updated by Peter Amstutz about 4 years ago
- Start date changed from 04/01/2020 to 05/01/2020
- Due date changed from 07/01/2020 to 08/01/2020
Updated by Peter Amstutz about 4 years ago
- Start date changed from 05/01/2020 to 08/01/2020
- Due date changed from 08/01/2020 to 11/30/2020
Updated by Peter Amstutz almost 4 years ago
- Start date changed from 08/01/2020 to 05/01/2020
- Due date changed from 11/30/2020 to 06/30/2020
Updated by Peter Amstutz almost 4 years ago
- Due date changed from 06/30/2020 to 07/31/2020
Updated by Peter Amstutz almost 4 years ago
- Start date changed from 05/01/2020 to 05/20/2020
Updated by Peter Amstutz almost 4 years ago
- Start date changed from 05/20/2020 to 06/03/2020
- Due date changed from 07/31/2020 to 08/31/2020
Updated by Peter Amstutz almost 4 years ago
- Start date changed from 06/03/2020 to 06/17/2020
- Due date changed from 08/31/2020 to 09/16/2020
Updated by Peter Amstutz almost 4 years ago
- Start date changed from 06/17/2020 to 07/29/2020
- Due date changed from 09/16/2020 to 11/11/2020
Updated by Peter Amstutz over 3 years ago
- Start date changed from 07/29/2020 to 10/01/2020
- Due date changed from 11/11/2020 to 01/31/2021
Updated by Peter Amstutz over 3 years ago
- Start date changed from 10/01/2020 to 01/01/2021
- Due date changed from 01/31/2021 to 04/30/2021
Updated by Peter Amstutz over 3 years ago
- Start date changed from 01/01/2021 to 04/01/2021
- Due date changed from 04/30/2021 to 07/31/2021
Updated by Peter Amstutz about 3 years ago
- Start date changed from 04/01/2021 to 07/01/2021
- Due date changed from 07/31/2021 to 11/30/2021
Updated by Peter Amstutz about 3 years ago
- Related to Idea #17348: Example workflow template which streams data from S3 in first step, does some computation steps, and uploads results back to S3. added
Updated by Peter Amstutz almost 3 years ago
- Start date changed from 07/01/2021 to 08/01/2021
- Due date changed from 11/30/2021 to 12/31/2021
Updated by Peter Amstutz over 2 years ago
- Start date changed from 08/01/2021 to 09/01/2021
Updated by Peter Amstutz over 2 years ago
- Start date changed from 09/01/2021 to 10/01/2021
- Due date changed from 12/31/2021 to 01/31/2022
Updated by Peter Amstutz over 2 years ago
- Start date changed from 10/01/2021 to 01/01/2022
- Due date changed from 01/31/2022 to 06/30/2022
Updated by Peter Amstutz over 2 years ago
- Start date changed from 01/01/2022 to 06/01/2022
- Due date changed from 06/30/2022 to 09/30/2022
Updated by Peter Amstutz almost 2 years ago
- Start date changed from 06/01/2022 to 08/01/2022
- Due date changed from 09/30/2022 to 11/30/2022
Updated by Peter Amstutz almost 2 years ago
- Start date changed from 08/01/2022 to 10/01/2022
- Due date changed from 11/30/2022 to 01/31/2023
Updated by Peter Amstutz over 1 year ago
- Start date changed from 10/01/2022 to 11/01/2022
- Due date changed from 01/31/2023 to 02/28/2023
Updated by Peter Amstutz about 1 year ago
- Due date changed from 02/28/2023 to 04/30/2023
Updated by Peter Amstutz about 1 year ago
- Start date changed from 11/01/2022 to 03/01/2023
- Due date changed from 04/30/2023 to 09/30/2023
Updated by Peter Amstutz about 1 year ago
- Start date changed from 03/01/2023 to 05/01/2023
- Due date changed from 09/30/2023 to 11/30/2023
Updated by Peter Amstutz 11 months ago
- Start date changed from 05/01/2023 to 09/01/2023
- Due date changed from 11/30/2023 to 12/31/2023
Updated by Peter Amstutz 11 months ago
- Start date changed from 09/01/2023 to 01/01/2024
- Due date changed from 12/31/2023 to 03/31/2024
Updated by Peter Amstutz 2 months ago
- Start date changed from 01/01/2024 to 01/01/2025
- Due date changed from 03/31/2024 to 03/31/2025
Actions