It's mysterious that the error text doesn't match the most obvious bit of code that would produce such an error, i.e., source:sdk/cli/bin/arv-run-pipeline-instance -- even old versions.

      debuglog "create job: #{j[:errors] rescue nil} with attributes #{body}", 0

      msg = "" 
      j[:errors].each do |err|
        msg += "Error creating job for component #{component}: #{err}\n" 
      end
      msg += "Job submission was: #{body.to_json}" 

      pipeline.log_stderr(msg)

Aside from that mystery, it seems like the fix for this is to have arv-run-pipeline-instance treat a timeout or 5xx response from the API as a transient error: follow the "can't make progress right now" code path instead of "fail pipeline".

• Is it worth writing a test for this issue?
• Taking into account #3005, would it be convenient to only retry a defined number of times? Retrying on any kind of exception seems too broad

Lucas Di Pentima wrote:

• Is it worth writing a test for this issue?

I wish, but given that crunch1's days are numbered and a-r-p-i doesn't have a good test setup, I'd say no.

• Taking into account #3005, would it be convenient to only retry a defined number of times? Retrying on any kind of exception seems too broad

I don't think it would be particularly convenient and I don't think it would be right either. For example, if the api server is down (even for a long time) I think the best behavior would be to keep trying until it comes up.

Perhaps worth noting that in general when a-r-p-i can't do anything it just exits without doing anything. The #3005 bug was that there are some error cases that indicate a pipeline is effectively deadlocked and should be cancelled. The problem here is just that the #3005 bugfix went too far and interpreted a transient network error as a sign of pipeline deadlock.

Thanks for the explanation, so this LGTM. Thanks.