Arvados

Lucas Di Pentima wrote:

On June 21st the addressable v2.8.7 gem was released. This new version requests public_suffix v6.0.0 that is not compatible with ruby 2.7.

• addressable 2.8.6 depends on public_suffix >= 2.0.2, < 6.0
• addressable 2.8.7 depends on public_suffix >= 2.0.2, < 7.0

It is still completely possible to install the gem on Ruby 2.7. The problem is that the gem install resolver only considers using the most recent version of everything, and pushes any more sophisticated resolution off to the user (hence the message telling you to say the version you want yourself).

Brett Smith wrote in #note-1:

It is still completely possible to install the gem on Ruby 2.7. The problem is that the gem install resolver only considers using the most recent version of everything, and pushes any more sophisticated resolution off to the user (hence the message telling you to say the version you want yourself).

Right, so in this particular case we could just fix the arvados-fomula by making it explicitly request the correct addressable gem version for arvados-cli and arvados-login-sync at least. Other packages requiring the arvados-google-api-client may also need some tweaking.

It would be better to pin public_suffix, since that's the gem that's actually incompatible with Ruby 2.7.

Doing this in arvados-formula would address the specific failure of test-provision-ubuntu2004. However, I believe in the past we have done pins like this directly in the source Gemfile or similar, because we have other things in our development stack that similarly run gem install arvados-cli. It's not ideal but putting the limit in the Gemfile means we only have to solve the problem once, vs. teaching each install process how to pin the specific versions it needs.

Discussed the plan with Tom in Matrix ahead of time and got sign-off. Given that, the implementation is too simple to review. But for posterity: 21931-public-suffix-pin @ 2f4fb1522c89c29a94854bf9f26fb6d13959f2d4

developer-run-tests: #4325

build-packages-multijob: #4284

test-provision: #913 - test-provision is still failing because it uses the published gem. build-packages shows that we got past the problem generally, so I expect this to succeed after we publish a new version of arvados-google-api-client.

Per Peter, updated the old build-publish-gem-arvados-google-api-client Jenkins job for the current gem build recipe.

build-publish-gem-arvados-google-api-client: #4

https://rubygems.org/gems/arvados-google-api-client/versions/0.8.7.7

test-provision: #914

This is turning into a "fix a dozen other things that are subtly broken" ticket.

After the last test-provision failed, I tried starting from build-packages-multijob again, since that kicks off the entire pipeline that ultimately includes test-provision.

build-packages-multijob: #4287

That yielded docker-jobs-image: #3253 which built:

12:29:11 Using /tmp/workspace/docker-jobs-image/sdk/python for version number calculation of /tmp/workspace/docker-jobs-image/sdk/python
12:29:11 Using /tmp/workspace/docker-jobs-image/sdk/cwl for version number calculation of /tmp/workspace/docker-jobs-image/sdk/cwl
12:29:12 cwl_runner_version 2.8.0.dev20240701191850 python_sdk_version 2.8.0.dev20240701191850
…
12:29:35 Successfully built 923a44547a65
12:29:35 Successfully tagged arvados/jobs:2.8.0.dev20240701191850

Now test-provision-ubuntu2004 succeeds but the other test-provision jobs are all failing because they're looking for a different version of the arvados/jobs image. For example, test-provision-debian12: #70 :

Running test CWL workflow
INFO /usr/bin/cwl-runner 2.8.0.dev20240703133808, arvados-python-client 2.8.0.dev20240703133808, cwltool 3.1.20240508115724
INFO Resolved 'hasher-workflow.cwl' to 'file:///tmp/cluster_tests/hasher-workflow.cwl'
INFO hasher-workflow.cwl:36:7: Unknown hint WorkReuse
INFO hasher-workflow.cwl:50:7: Unknown hint WorkReuse
INFO hasher-workflow.cwl:64:7: Unknown hint WorkReuse
fatal: not a git repository (or any of the parent directories): .git
INFO Using cluster dbn12 (https://dbn12.local:443/)
INFO Upload local files: "test.txt" 
INFO Uploaded to f55e750025853f5b8ccae3ca79240f65+54 (dbn12-4zz18-eg5ncx3wseqdi96)
INFO Uploading workflow dependencies
Error: No such object: arvados/jobs:2.8.0.dev20240703133808
INFO ['docker', 'pull', 'arvados/jobs:2.8.0.dev20240703133808']
Error response from daemon: manifest for arvados/jobs:2.8.0.dev20240703133808 not found: manifest unknown: manifest unknown
ERROR Workflow error:
Command '['docker', 'pull', 'arvados/jobs:2.8.0.dev20240703133808']' returned non-zero exit status 1.
Traceback (most recent call last):
  File "/usr/lib/python3-arvados-cwl-runner/lib/python3.9/site-packages/arvados_cwl/arvdocker.py", line 124, in arv_docker_get_image
    result = dockerjob.get_image(dockerRequirement, pull_image,
  File "/usr/lib/python3-arvados-cwl-runner/lib/python3.9/site-packages/cwltool/docker.py", line 149, in get_image
    subprocess.check_call(cmd, stdout=sys.stderr)  # nosec
  File "/usr/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker', 'pull', 'arvados/jobs:2.8.0.dev20240703133808']' returned non-zero exit status 1.

So I have to figure out why they're not on the same page, either with docker-jobs-image and/or with test-provision-ubuntu2004. Current top theory:

• build-packages-multijob has still been uploading packages for other distros as branches get merged to main
• but docker-jobs-image has not run for any of those newer merges because none of those build-packages-multijob runs succeeded because of this Ubuntu 20.04 blocker
• test-provision installs packages from the -dev repository and uses them to figure out which version of arvados/jobs it should get

I could work around by forcing a run of docker-jobs-image from current main I guess.

Brett Smith wrote in #note-9:

• build-packages-multijob has still been uploading packages for other distros as branches get merged to main
• but docker-jobs-image has not run for any of those newer merges because none of those build-packages-multijob runs succeeded because of this Ubuntu 20.04 blocker
• test-provision installs packages from the -dev repository and uses them to figure out which version of arvados/jobs it should get

Confirmed. This behavior comes from a-c-r:

arvados_cwl/__init__.py:    executor.loadingContext.default_docker_image = arvargs.submit_runner_image or "arvados/jobs:"+__version__

So for non-ubuntu2004 distros, test-provision installs a version of a-c-r from a recent merge to main (specifically 853867e2f48fc4c934071e704c539b2aae904537), but there's no matching arvados/jobs image because that build has been blocked by the failure of build-packages-ubuntu2004. Forcing a matching arvados/jobs build should resolve the issue.

docker-jobs-image: #3254

test-provision: #918

Now that everything is green, the branch is merged.

2.7.4 should cherry-pick af3ef3ca4b4df2c775eeacef7791a09333bec586. The other commit in the branch 2f4fb1522c89c29a94854bf9f26fb6d13959f2d4 was required to keep Ubuntu 20.04 working after other changes in main. It's effectively reverting one gem upgrade. That gem was never upgraded in 2.7, so this commit wouldn't be meaningful there.

For the record: it seems there's nothing to cherry-pick for 2.7 because ruby-google-api-client is not incorporated into the 2.7 source tree, but since the 0.8.7.7 gem is published, and the gemspec says s.add_dependency('arvados-google-api-client', '>= 0.8.7.3', '< 0.8.9') I don't think there is anything to do.

So potential dependency issues in the arvados 2.7 ruby gem are solved externally to the 2.7-release branch by releasing a new arvados-google-api-client gem version.