Arvados

I think it would be a good idea to be able to change the default for the resubmit-non-preemptible behavior.

One option is to just switch the default in arvados-cwl-runner going forward.

Another option would be to add a configuration item to the config file, publish that, and have arvados-cwl-runner use it.

I think there's two principals in tension here.

There's the idea that we shouldn't do anything that the user didn't explicitly ask for. Resubmitting the job non-preemptible arguably counts, because the user asked to use preemptible nodes. I feel like we also talk ourselves into corners here, because users don't know these features exist to be able to turn them on.

On the other hand, what the user probably actually wants is their job to finish. This feature is intended to balance the cost savings of spot instances with the reliability of on-demand instances. Spot instances have no purpose except to save money, so forcing jobs to be run on spot instances when the return on investment is likely negative (because it takes longer, is less reliable, and/or costs more) is counter productive.