Arvados

We should figure out how many times each block is fetched during this job. For example, if the data is being read in an inconvenient order and the Python SDK block cache isn't doing well, it's conceivable each block is being transmitted over the network (say) 20 times. If all the data is stuck on one keepstore server, that would be ~250 MiB/s on the storage node, which would be reasonable. If this is what's happening:

• perhaps the job is pathological (filenames sorted differently when reading than when writing, e.g., 1,2,3 on the way in, 1,10,100,11 on the way out?)
• perhaps the python SDK block cache is buggy?
• perhaps we need #3640

Keepstore disk → 128 clients should certainly sustain way more than 12.8 MiB/s on su92l.

This is a very noisy experiment, enough so that I wonder how much it tells us about any one component.

The job is a shell script that calls a second shell script that's basically a utility wrapper around the Python SDK. Here's a list of all the shell commands in the job script that make API requests, and what those are:

• arv-dax setup: Get the current task
• arv-dax script_parameters: Get the current job
• arv-dax task sequence: Get the current task
• arv-dax task parameters: Get the current task
• find: Get the 11MiB stress directory collection (via FUSE)
• arv-dax task finish: Create a collection; get the current task; update the current task

You could literally go from eight API calls to four just by fetching the current task once, and fetching data from its individual fields as needed.

It's also possible that the discovery document isn't being cached between these commands. If that's the case, then the number of API requests is almost doubled: each item in the list above needs to fetch the discovery document, then make its request(s).

The job has 16 * 8 == 128 tasks in flight simultaneously most of the time. We already know that API server response times degrade noticeably as request parallelism goes up; see #5901. I think it's very possible that the API server is at least as much your problem as Keep itself, if not more.

I don't deny that people are going to write job scripts like this, or that this job shouldn't run faster. But from this test, I think it's premature to conclude that there's a major problem with Keep performance.

Brett Smith wrote:

It's also possible that the discovery document isn't being cached between these commands. If that's the case, then the number of API requests is almost doubled: each item in the list above needs to fetch the discovery document, then make its request(s).

This was bad speculation. crunch-job sets $HOME to be the same as $TASK_WORK, so even a job running inside a Docker image can write here no problem.