Arvados

Container priority is inherited from the root container. When we have a bunch of independent workflows, this means workflows submitted earlier, and all their children, will get priority over ones submitted later, so they should generally complete first. The steps themselves have the same priority and it doesn't matter that much what order they run in.

However, when we have a workflow runner that submits workflow runners, and the work steps are 3rd level, they all have the same priority. As a result, we lose the behavior of doing work for earlier sub-workflows before processing later workflows.

I think we want to adjust the inherited priority based on the created_at time so that at the 2nd level, steps tend to execute in order they are submitted, and at the 3rd level, steps owned by earlier sub-workflows run before steps owned by later sub-workflows.

Currently, the priority is essentially sliced into 14 bits for the toplevel priority (1-1000) and 50 bits based on created_at (down to the millisecond).

To get perfect ordering to 3rd level, we'd need to slice this up into more levels, e.g.

10 bits for toplevel, 17 bits for second level, 17 bits for second level, 17 bits for 3rd level

but maybe we can figure out another way to do it.