Feature #3980

[SDKs] CLI tool to view real time logs from running jobs/pipelines, much like the Log tab of the Workbench PipelineInstance#show page

Added by Abram Connelly almost 5 years ago. Updated over 3 years ago.

Status:
New
Priority:
Normal
Assigned To:
-
Category:
SDKs
Target version:
Start date:
09/24/2014
Due date:
% Done:

0%

Estimated time:
Story points:
1.0

Description

It would be nice to have a command line tool that was able to receive log information from currently running jobs so that it could be monitored from the command line rather than going through the 'Log' page in workbench.

This should look just like crunch-job in local mode, or the Log tab of a running job in Workbench: no json formatting, etc., just the text.

For pipeline instances, this should be implemented as arv-run-pipeline-instance --log-only --instance {uuid} because a-r-p-i will need to make use of websockets soon enough anyway (but combining --log with --run-*-here is a different story, #4503).

Something like this should make it easy to share code between --log-only and the "don't poll" part of #4503:

def each_pipeline_state_change(my_instance_uuid)
  # yield when pipeline changes state. return when pipeline has finished
  subscribe_to_websocket my_instance_uuid
  listen_to_websocket do |msg|
    if {msg is a pipelineInstance update event with uuid==my_instance_uuid}
      if {new instance has different job uuids, job states, etc. than last time}
        subscribe_to_websocket {any new job uuids that have appeared}
        yield new_instance_properties
      end
      if {pipeline is not running}
        return
      end
    elsif {logging mode is enabled} and {msg is a stderr event for a job in this instance}
      print msg.properties.text
    end
  end
end

For the case of a single job, this could just as well be a standalone ruby program.

arv tail {uuid} should cause arv to invoke the appropriate program, depending on whether {uuid} is a pipeline_instance or a job uuid.


Related issues

Blocks Arvados - Feature #4503: [SDKs] Option to watch logs (instead of exiting) when queuing a job or pipeline instance using arv job create, arv pipeline run, etc.New

History

#1 Updated by Tom Clegg almost 5 years ago

  • Subject changed from CLI tool to view real time logs from running jobs/pipelines to [SDKs] CLI tool to view real time logs from running jobs/pipelines, much like the Log tab of the Workbench PipelineInstance#show page
  • Story points set to 1.0

#2 Updated by Tom Clegg almost 5 years ago

  • Target version set to Arvados Future Sprints

#3 Updated by Tom Clegg over 4 years ago

  • Description updated (diff)
  • Category set to SDKs

#4 Updated by Brett Smith over 3 years ago

Note that the functionality was done in #3603 (invoked as arv ws --pipeline or arv ws --job), but I'll leave this open for now since it covers some interface improvements.

Also available in: Atom PDF