Arvados

Mar 18 03:53:48 ip-172-25-144-184 arvados-dispatch-cloud[283002]: {"ClusterID":"xxxxx","InstanceType":"r52xlarge.preemptible","PID":283002,"error":"InvalidParameterValue: Not enough free addresses in subnet subnet-0f83ca79\n\tstatus code: 400, request id: 6cbcffe1-5b77-4dee-8fbf-c20f67892c95","level":"error","msg":"create failed","time":"2024-03-18T03:53:48.927972989Z"}

This is a subnet-specific error (it should switch to the other subnet) but the current function won't recognize it as such:

func isErrorSubnetSpecific(err error) bool {
    aerr, ok := err.(awserr.Error)
    if !ok {
        return false
    }
    code := aerr.Code()
    return strings.Contains(code, "Subnet") ||
        code == "InsufficientInstanceCapacity" ||
        code == "InsufficientVolumeCapacity" ||
        code == "Unsupported" 
}

Because the error was unrecognized, it seems the fallback behavior seems to be to rate limit itself by setting maximum concurrent containers.