Project

General

Profile

Actions
Copy link

Task #3309

open

Idea #3511: [Documentation] Present an efficient pattern for developing a pipeline template with multiple crunch scripts

[Documentation] Present an efficient pattern for developing a Python program as a Crunch job

Added by Tom Clegg almost 10 years ago. Updated about 7 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Target version:
-

Description

Approximate outline:
  • Find/make a small test dataset and put it in Keep
  • Clone git repository to shell VM (or start a new one)
  • Put crunch wrapper in /crunch_scripts/foobar
  • Run on VM in terminal using crunch-job
  • Repeat until this works
  • Commit and push
  • Run as a real job using small test dataset
  • Repeat until this works
  • Run as a real job using real data
Simplifying assumptions:
  • One task per job

Here, just present the minimum feature set. Refer to appropriate doc sections for related features, something like "Learn more about what scripts are expected to see/do/output on the {foobar doc page}" (link to outcome of #3044)

Related issues

Related to Arvados - Idea #3308: [Documentation] Present an efficient pattern for developing a C program using CrunchClosedActions
Actions
Copy link

Also available in: Atom PDF