Arvados

This started after merging #21230 and #21394.

Running test CWL workflow
INFO /usr/bin/cwl-runner 2.8.0.dev20240119143735, arvados-python-client 2.8.0.dev20240119143735, cwltool 3.1.20230601100705
INFO Resolved 'hasher-workflow.cwl' to 'file:///tmp/cluster_tests/hasher-workflow.cwl'
INFO hasher-workflow.cwl:36:7: Unknown hint WorkReuse
INFO hasher-workflow.cwl:50:7: Unknown hint WorkReuse
INFO hasher-workflow.cwl:64:7: Unknown hint WorkReuse
INFO Using cluster dbn11 (https://dbn11.local:443/)
DEBUG ENTER upload_job_order 1705679832.0055475
DEBUG ENTER scandeps 1705679832.007659
DEBUG EXIT scandeps 1705679832.0088325 0.0011734962463378906
DEBUG ENTER collect uuids 1705679832.0089567
DEBUG EXIT collect uuids 1705679832.0090437 8.702278137207031e-05
DEBUG ENTER collect uploads 1705679832.0090969
DEBUG EXIT collect uploads 1705679832.0091777 8.082389831542969e-05
DEBUG ENTER fetch_uuids 1705679832.0092268
DEBUG EXIT fetch_uuids 1705679832.0092647 3.790855407714844e-05
DEBUG ENTER mapper 1705679832.010263
INFO Upload local files: "test.txt" 
INFO Uploaded to f55e750025853f5b8ccae3ca79240f65+54 (dbn11-4zz18-efp6b3h7f97ysul)
DEBUG EXIT mapper 1705679832.0954368 0.0851738452911377
DEBUG ENTER collectloc 1705679832.0955882
DEBUG EXIT collectloc 1705679832.0957391 0.00015091896057128906
DEBUG ENTER setloc 1705679832.0959854
DEBUG EXIT setloc 1705679832.096145 0.0001595020294189453
DEBUG EXIT upload_job_order 1705679832.0962608 0.0907132625579834
INFO Uploading workflow dependencies
DEBUG ENTER upload_workflow_deps 1705679832.096431
DEBUG ENTER upload_docker 1705679832.096509
INFO ['docker', 'pull', 'arvados/jobs:2.8.0.dev20240119143735']
2.8.0.dev20240119143735: Pulling from arvados/jobs
[`docker pull` progress omitted]
Digest: sha256:f7991a0df2f3dd11b51a3b2aaba0ceb6af5c61195f5671daca6e963d4d8aa487
Status: Downloaded newer image for arvados/jobs:2.8.0.dev20240119143735
docker.io/arvados/jobs:2.8.0.dev20240119143735
INFO Uploading Docker image arvados/jobs:2.8.0.dev20240119143735
2024-01-19 15:57:30 arvados.arv_put[41618] INFO: Creating new cache file at /root/.cache/arvados/arv-put/f8be69c1db20b934481409a3f508ad24
2024-01-19 15:57:32 arvados.arv_put[41618] INFO: Collection saved as 'Docker image arvados jobs:2.8.0.dev20240119143735 sha256:b3d59'
dbn11-4zz18-cky0zrtpcygez8q
2024-01-19 15:57:32 arvados.cwl-runner.metrics[41618] DEBUG: EXIT upload_docker 1705679852.4451454 20.348636388778687 (X-Request-Id: req-z32hupda8gmghtycmu3f)
2024-01-19 15:57:32 arvados.cwl-runner.metrics[41618] DEBUG: EXIT upload_workflow_deps 1705679852.445314 20.348882913589478
2024-01-19 15:57:32 cwltool[41618] ERROR: Workflow error:
"filename 'b3d596985843bccad4c72223479cb9b8159cf6483fd0bfbe732ef152c10e7fcd.json' not found" 
Traceback (most recent call last):
  File "/usr/share/python3.9/dist/python3-arvados-cwl-runner/lib/python3.9/site-packages/arvados_cwl/arvdocker.py", line 141, in arv_docker_get_image
    arvados.commands.keepdocker.main(args, stdout=sys.stderr, install_sig_handlers=False, api=api_client)
  File "/usr/share/python3.9/dist/python3-arvados-cwl-runner/lib/python3.9/site-packages/arvados/commands/keepdocker.py", line 546, in main
    json_file = image_tar.extractfile(image_tar.getmember(json_filename))
  File "/usr/lib/python3.9/tarfile.py", line 1790, in getmember
    raise KeyError("filename %r not found" % name)
KeyError: "filename 'b3d596985843bccad4c72223479cb9b8159cf6483fd0bfbe732ef152c10e7fcd.json' not found" 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/python3.9/dist/python3-arvados-cwl-runner/lib/python3.9/site-packages/cwltool/main.py", line 1301, in main
    (out, status) = real_executor(
  File "/usr/share/python3.9/dist/python3-arvados-cwl-runner/lib/python3.9/site-packages/arvados_cwl/executor.py", line 695, in arv_executor
    merged_map = upload_workflow_deps(self, tool, runtimeContext)
  File "/usr/share/python3.9/dist/python3-arvados-cwl-runner/lib/python3.9/site-packages/arvados_cwl/runner.py", line 741, in upload_workflow_deps
    upload_docker(arvrunner, tool, runtimeContext)
  File "/usr/share/python3.9/dist/python3-arvados-cwl-runner/lib/python3.9/site-packages/arvados_cwl/runner.py", line 557, in upload_docker
    upload_docker(arvrunner, s.embedded_tool, runtimeContext)
  File "/usr/share/python3.9/dist/python3-arvados-cwl-runner/lib/python3.9/site-packages/arvados_cwl/runner.py", line 551, in upload_docker
    arvados_cwl.arvdocker.arv_docker_get_image(arvrunner.api, docker_req, True, runtimeContext)
  File "/usr/share/python3.9/dist/python3-arvados-cwl-runner/lib/python3.9/site-packages/arvados_cwl/arvdocker.py", line 157, in arv_docker_get_image
    cached_lookups[dockerRequirement["dockerImageId"]] = pdh
  File "schema_salad/sourceline.py", line 250, in __exit__
cwltool.errors.WorkflowException: "filename 'b3d596985843bccad4c72223479cb9b8159cf6483fd0bfbe732ef152c10e7fcd.json' not found" 
Build step 'Execute shell' marked build as failure