Arvados

Error messages in build-package-* pipelines were like:

...
21:12:38 ======= Start upload packages
21:12:38 /usr/local/arvados-dev/jenkins/run_upload_packages.py --repo dev -H jenkinsapt@apt.arvados.org -o Port=2222 --workspace /tmp/workspace/build-packages-debian10 debian10
21:12:38 Not adding keepproxy_2.6.0~dev20230127195601-1_amd64.deb, it is already present in repo buster-dev
21:12:42 Not adding python3-crunchstat-summary_2.6.0~dev20230118151106-1_amd64.deb, it is already present in repo buster-dev
21:12:42 Not adding keepstore_2.6.0~dev20230127195601-1_amd64.deb, it is already present in repo buster-dev
...
21:12:44 Loading packages...
21:12:44 Generating metadata files and linking package files...
21:12:44 ERROR: unable to publish: unable to process packages: error linking file to /data/var-www/aptly_public/buster/pool/main/p/python-arvados-cwl-runner/python-arvados-cwl-runner_2.5.0-1_all.deb: file already exists and is different
21:12:49 Traceback (most recent call last):
21:12:49   File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 356, in <module>
21:12:49     main(sys.argv[1:])
21:12:49   File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 352, in main
21:12:49     build_suite_and_upload(target, last_upload_ts, args)
21:12:49   File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 342, in build_suite_and_upload
21:12:49     suite.update_packages(since_timestamp)
21:12:49   File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 115, in update_packages
21:12:49     self.post_uploads(upload_paths)
21:12:49   File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 237, in post_uploads
21:12:49     *self._paths_basenames(paths))
21:12:49   File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 195, in _run_script
21:12:49     self.__class__.__name__, *(pipes.quote(s) for s in args)))
21:12:49   File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
21:12:49     raise CalledProcessError(retcode, cmd)
21:12:49 subprocess.CalledProcessError: Command '['ssh', '-oPort=2222', '-q', 'jenkinsapt@apt.arvados.org', 'bash', '-ec', '\'\ncd "$1"; shift\nDISTNAME=$1; shift\nfor package in "$@"; do\n  set +e\n  aptly repo search "$DISTNAME" "${package%.deb}" >/dev/null 2>&1\n  RET=$?\n  set -e\n  if [[ $RET -eq 0 ]]; then\n    echo "Not adding $package, it is already present in repo $DISTNAME"\n    rm "$package"\n  else\n    aptly repo add -remove-files "$DISTNAME" "$package"\n  fi\ndone\naptly publish update "$DISTNAME" filesystem:"${DISTNAME%-*}":\n\'', 'DebianPackageSuite', 'tmp/debian10', 'buster-dev', "'keepproxy_2.6.0~dev20230127195601-1_amd64.deb'", "'python3-crunchstat-summary_2.6.0~dev20230118151106-1_amd64.deb'", "'keepstore_2.6.0~dev20230127195601-1_amd64.deb'", "'crunch-dispatch-local_2.6.0~dev20230127195601-1_amd64.deb'", 'python-crunchstat-summary_2.6.0.dev20230118151106-1_all.deb', 'python-cwltest_2.3.20230109094201-1_all.deb', 'python-arvados-python-client_2.6.0.dev20230118151106-1_all.deb', "'keep-balance_2.6.0~dev20230127195601-1_amd64.deb'", "'keep-exercise_2.6.0~dev20230127195601-1_amd64.deb'", "'crunch-run_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-dispatch-cloud_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-docker-cleaner_2.3.0~dev20210729201354-1_amd64.deb'", "'arvados-sync-groups_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-workbench_2.6.0~dev20230127195601-1_amd64.deb'", 'python-arvados-fuse_2.6.0.dev20230118151106-1_all.deb', "'keep-block-check_2.6.0~dev20230127195601-1_amd64.deb'", "'libpam-arvados-go_2.6.0~dev20230127195601-1_amd64.deb'", "'keep-rsync_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-sync-users_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-server_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-client_2.6.0~dev20230127195601-1_amd64.deb'", 'python-arvados-docker-cleaner_2.3.0.dev20210729201354-1_all.deb', "'arvados-src_2.6.0~dev20230127195636-1_all.deb'", "'crunchstat_2.6.0~dev20230127195601-1_amd64.deb'", "'python3-arvados-fuse_2.6.0~dev20230118151106-1_amd64.deb'", "'arvados-health_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-controller_2.6.0~dev20230127195601-1_amd64.deb'", "'crunch-dispatch-slurm_2.6.0~dev20230127195601-1_amd64.deb'", "'python3-arvados-python-client_2.6.0~dev20230118151106-1_amd64.deb'", "'python3-arvados-cwl-runner_2.6.0~dev20230126192339-1_amd64.deb'", 'python-arvados-cwl-runner_2.6.0.dev20230126192339-1_all.deb', "'arvados-api-server_2.6.0~dev20230127195636-1_amd64.deb'", 'python-arvados-user-activity_2.6.0.dev20230118151106-1_all.deb', "'keep-web_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-dispatch-lsf_2.6.0~dev20230127195601-1_amd64.deb'", 'python3-cwltest_2.3.20230109094201-1_amd64.deb', "'arvados-ws_2.6.0~dev20230127195601-1_amd64.deb'", "'python3-arvados-user-activity_2.6.0~dev20230118151106-1_amd64.deb'", "'arvados-git-httpd_2.6.0~dev20230127195601-1_amd64.deb'"]' returned non-zero exit status 1.
21:12:49 ======= upload packages -- FAILED
...

Removed all non-dev 2.5.0-1 packages from the *-dev repositories using aptly commands, and then ran aptly publish update on them to double check everything was ok. Jenkins build-packages-* pipelines started to work again.

Next step, find out where on our script collection this issue was generated so we can fix it.

Updates at 46883d9 - branch 19937-build-packages-fix

• Fixes some red herring error logging on the test packages script.
• Makes version-at-commit.sh only output development versions.
• Fixes a previously existing issue on version-at-commit.sh that made it output ever changing dev versions on old commits from release branches where their nearest tag is older than their merge base commits.

The below table shows the changes from the current vs new versions of the script.

LGTM, thanks. The table in #note-9 is very helpful / reassuring!