https://dev.arvados.org/https://dev.arvados.org/favicon.ico?15576888422023-01-25T18:11:26ZArvadosArvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1102702023-01-25T18:11:26ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>To be scheduled</i> to <i>2023-02-15 sprint</i></li></ul> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1103612023-01-27T16:44:11ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>2023-02-15 sprint</i> to <i>2023-02-01 sprint</i></li></ul> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1103622023-01-27T16:44:44ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1103632023-01-27T16:45:34ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Assigned To</strong> set to <i>Lucas Di Pentima</i></li></ul> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1104452023-01-30T18:54:52ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Error messages in <code>build-package-*</code> pipelines were like:</p>
<pre>
...
21:12:38 ======= Start upload packages
21:12:38 /usr/local/arvados-dev/jenkins/run_upload_packages.py --repo dev -H jenkinsapt@apt.arvados.org -o Port=2222 --workspace /tmp/workspace/build-packages-debian10 debian10
21:12:38 Not adding keepproxy_2.6.0~dev20230127195601-1_amd64.deb, it is already present in repo buster-dev
21:12:42 Not adding python3-crunchstat-summary_2.6.0~dev20230118151106-1_amd64.deb, it is already present in repo buster-dev
21:12:42 Not adding keepstore_2.6.0~dev20230127195601-1_amd64.deb, it is already present in repo buster-dev
...
21:12:44 Loading packages...
21:12:44 Generating metadata files and linking package files...
21:12:44 ERROR: unable to publish: unable to process packages: error linking file to /data/var-www/aptly_public/buster/pool/main/p/python-arvados-cwl-runner/python-arvados-cwl-runner_2.5.0-1_all.deb: file already exists and is different
21:12:49 Traceback (most recent call last):
21:12:49 File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 356, in <module>
21:12:49 main(sys.argv[1:])
21:12:49 File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 352, in main
21:12:49 build_suite_and_upload(target, last_upload_ts, args)
21:12:49 File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 342, in build_suite_and_upload
21:12:49 suite.update_packages(since_timestamp)
21:12:49 File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 115, in update_packages
21:12:49 self.post_uploads(upload_paths)
21:12:49 File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 237, in post_uploads
21:12:49 *self._paths_basenames(paths))
21:12:49 File "/usr/local/arvados-dev/jenkins/run_upload_packages.py", line 195, in _run_script
21:12:49 self.__class__.__name__, *(pipes.quote(s) for s in args)))
21:12:49 File "/usr/lib/python3.7/subprocess.py", line 347, in check_call
21:12:49 raise CalledProcessError(retcode, cmd)
21:12:49 subprocess.CalledProcessError: Command '['ssh', '-oPort=2222', '-q', 'jenkinsapt@apt.arvados.org', 'bash', '-ec', '\'\ncd "$1"; shift\nDISTNAME=$1; shift\nfor package in "$@"; do\n set +e\n aptly repo search "$DISTNAME" "${package%.deb}" >/dev/null 2>&1\n RET=$?\n set -e\n if [[ $RET -eq 0 ]]; then\n echo "Not adding $package, it is already present in repo $DISTNAME"\n rm "$package"\n else\n aptly repo add -remove-files "$DISTNAME" "$package"\n fi\ndone\naptly publish update "$DISTNAME" filesystem:"${DISTNAME%-*}":\n\'', 'DebianPackageSuite', 'tmp/debian10', 'buster-dev', "'keepproxy_2.6.0~dev20230127195601-1_amd64.deb'", "'python3-crunchstat-summary_2.6.0~dev20230118151106-1_amd64.deb'", "'keepstore_2.6.0~dev20230127195601-1_amd64.deb'", "'crunch-dispatch-local_2.6.0~dev20230127195601-1_amd64.deb'", 'python-crunchstat-summary_2.6.0.dev20230118151106-1_all.deb', 'python-cwltest_2.3.20230109094201-1_all.deb', 'python-arvados-python-client_2.6.0.dev20230118151106-1_all.deb', "'keep-balance_2.6.0~dev20230127195601-1_amd64.deb'", "'keep-exercise_2.6.0~dev20230127195601-1_amd64.deb'", "'crunch-run_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-dispatch-cloud_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-docker-cleaner_2.3.0~dev20210729201354-1_amd64.deb'", "'arvados-sync-groups_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-workbench_2.6.0~dev20230127195601-1_amd64.deb'", 'python-arvados-fuse_2.6.0.dev20230118151106-1_all.deb', "'keep-block-check_2.6.0~dev20230127195601-1_amd64.deb'", "'libpam-arvados-go_2.6.0~dev20230127195601-1_amd64.deb'", "'keep-rsync_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-sync-users_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-server_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-client_2.6.0~dev20230127195601-1_amd64.deb'", 'python-arvados-docker-cleaner_2.3.0.dev20210729201354-1_all.deb', "'arvados-src_2.6.0~dev20230127195636-1_all.deb'", "'crunchstat_2.6.0~dev20230127195601-1_amd64.deb'", "'python3-arvados-fuse_2.6.0~dev20230118151106-1_amd64.deb'", "'arvados-health_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-controller_2.6.0~dev20230127195601-1_amd64.deb'", "'crunch-dispatch-slurm_2.6.0~dev20230127195601-1_amd64.deb'", "'python3-arvados-python-client_2.6.0~dev20230118151106-1_amd64.deb'", "'python3-arvados-cwl-runner_2.6.0~dev20230126192339-1_amd64.deb'", 'python-arvados-cwl-runner_2.6.0.dev20230126192339-1_all.deb', "'arvados-api-server_2.6.0~dev20230127195636-1_amd64.deb'", 'python-arvados-user-activity_2.6.0.dev20230118151106-1_all.deb', "'keep-web_2.6.0~dev20230127195601-1_amd64.deb'", "'arvados-dispatch-lsf_2.6.0~dev20230127195601-1_amd64.deb'", 'python3-cwltest_2.3.20230109094201-1_amd64.deb', "'arvados-ws_2.6.0~dev20230127195601-1_amd64.deb'", "'python3-arvados-user-activity_2.6.0~dev20230118151106-1_amd64.deb'", "'arvados-git-httpd_2.6.0~dev20230127195601-1_amd64.deb'"]' returned non-zero exit status 1.
21:12:49 ======= upload packages -- FAILED
...
</pre>
<p>Removed all non-dev <code>2.5.0-1</code> packages from the <code>*-dev</code> repositories using <code>aptly</code> commands, and then ran <code>aptly publish update</code> on them to double check everything was ok. Jenkins <code>build-packages-*</code> pipelines started to work again.</p>
<p>Next step, find out where on our script collection this issue was generated so we can fix it.</p> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1105462023-02-01T15:49:54ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2023-02-01 sprint</i> to <i>2023-02-15 sprint</i></li></ul> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1124202023-02-15T15:58:16ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2023-02-15 sprint</i> to <i>2023-03-01 sprint</i></li></ul> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1128572023-03-02T20:08:50ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Target version</strong> changed from <i>2023-03-01 sprint</i> to <i>Development 2023-03-15 sprint</i></li></ul> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1132242023-03-14T18:06:21ZLucas Di Pentimalucas.dipentima@curii.com
<ul></ul><p>Updates at <a class="changeset" title="19937: Only output development versions. Also, fixed a previously existing issue where a release..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/46883d9df0b15d2114192a5f5bc738c6735d3ec0">46883d9</a> - branch <code>19937-build-packages-fix</code></p>
<ul>
<li>Fixes some red herring error logging on the test packages script.</li>
<li>Makes <code>version-at-commit.sh</code> only output development versions.</li>
<li>Fixes a previously existing issue on <code>version-at-commit.sh</code> that made it output ever changing dev versions on old commits from release branches where their nearest tag is older than their merge base commits.</li>
</ul>
<p>The below table shows the changes from the current vs new versions of the script.</p>
<table>
<tr>
<th>Commit ID</th>
<th>Tag</th>
<th>Comment</th>
<th>Current Version</th>
<th>New Version</th>
</tr>
<tr>
<td><a class="changeset" title="Merge branch '19972-go-client-retry' closes #19972 Arvados-DCO-1.1-Signed-off-by: Tom Clegg <to..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f94ac6e8ad9aec3c781cd71b72fcc5e2c1cedd8d">f94ac6e8a</a></td>
<td></td>
<td>Current main’s HEAD</td>
<td>2.6.0~dev20230313145740</td>
<td>2.6.0~dev20230313145740</td>
</tr>
<tr>
<td><a class="changeset" title="Merge branch '19844-paging' refs #19844 Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amst..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/db166b9539a0f18621e2c980137d082357b1a680">db166b9</a></td>
<td>2.5.0</td>
<td>Located in main branch</td>
<td>2.5.0</td>
<td><span style="background:lightgreen;">2.5.0~dev20221220151318</span></td>
</tr>
<tr>
<td><a class="changeset" title="Merge branch '19880-wb1-pw-login-fix' refs #19880 Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/9b3344f612fcf11fb868e67bc5f0636160900520">9b3344f</a></td>
<td></td>
<td>Previous commit from 2.5.0 tag</td>
<td>2.5.0~dev20221216221833</td>
<td>2.5.0~dev20221216221833</td>
</tr>
<tr>
<td><a class="changeset" title="Update version numbers refs #19647 Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.amstutz@..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/5203a7e1ad323adf0d0bc5dfbc6c28045ef4b263">5203a7e</a></td>
<td></td>
<td>Subsequent commit from 2.5.0 tag</td>
<td>2.6.0~dev20221222191144</td>
<td>2.6.0~dev20221222191144</td>
</tr>
<tr>
<td><a class="changeset" title="Add 2.4.4 to upgrading notes. refs #19717 Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <peter.a..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f24931231cf80caa65f05800245922f3d7c8af2b">f249312</a></td>
<td>2.4.4</td>
<td>Located in 2.4-release branch</td>
<td>2.4.4</td>
<td><span style="background:lightgreen;">2.4.5~dev20221118143500</span></td>
</tr>
<tr>
<td><a class="changeset" title="Merge branch '19692-quieter-runtime-status' refs #19692 Arvados-DCO-1.1-Signed-off-by: Peter Ams..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/88cb0a31936d8a70ca811ccd3f4172b7346e712c">88cb0a3</a></td>
<td></td>
<td>Previous commit from 2.4.4 tag</td>
<td> 2.4.4~dev20221115145131</td>
<td>2.4.4~dev20221115145131</td>
</tr>
<tr>
<td><a class="changeset" title="Add upgrade note about change in default result order. refs #18943 Arvados-DCO-1.1-Signed-off-by..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/74057d05e57dc6833b330aeffffe966a698118ea">74057d0</a></td>
<td>2.4.0</td>
<td>Located in 2.4-release branch</td>
<td>2.4.0</td>
<td><span style="background:lightgreen;">2.4.1~dev20220405185643</span></td>
</tr>
<tr>
<td><a class="changeset" title="18865: Cherry-pick '18865-collection-permission-links' merge 3c73e2337ed73cd44e9bcc2d38a4dd74263..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f2861b559f7aec8dccc7d76705f15e4feb871848">f2861b5</a></td>
<td></td>
<td>Previous commit from 2.4.0 tag, in 2.4-release branch</td>
<td><span style="background:yellow;">2.6.0~dev20220405154430</span></td>
<td><span style="background:lightgreen;">2.4.0~dev20220405154430</span></td>
</tr>
<tr>
<td><a class="changeset" title="Merge branch '18941-arv-prefetch' refs #18941 Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pete..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/f41534fcdfa95c282a04d64158f8a7d850637768">f41534f</a></td>
<td></td>
<td>Commit from where 2.4-release branched off main</td>
<td>2.4.0~dev20220330193024</td>
<td>2.4.0~dev20220330193024</td>
</tr>
<tr>
<td><a class="changeset" title="Merge branch '18260-update-cwltool' refs #18260 Arvados-DCO-1.1-Signed-off-by: Peter Amstutz <pe..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/64194cf50545e1b6894b45cc18edbefbc3ac8707">64194cf</a></td>
<td></td>
<td>Commit from where 2.3-release branched off main</td>
<td>2.3.0~dev20211020182823</td>
<td>2.3.0~dev20211020182823</td>
</tr>
<tr>
<td><a class="changeset" title="Pin arvados 2.3.0 version in provision.sh. Refs #18130 Arvados-DCO-1.1-Signed-off-by: Lucas Di P..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/a598bba05ec26c8103bd7f8cf9e4dcf451469531">a598bba</a></td>
<td></td>
<td>First commit of 2.3-release branch</td>
<td><span style="background:yellow;">2.6.0~dev20211020193505</span></td>
<td><span style="background:lightgreen;">2.3.0~dev20211020193505</span></td>
</tr>
</table> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1132812023-03-15T15:34:21ZPeter Amstutzpeter.amstutz@curii.com
<ul><li><strong>Target version</strong> changed from <i>Development 2023-03-15 sprint</i> to <i>Development 2023-03-29 Sprint</i></li></ul> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1133222023-03-16T14:44:46ZTom Cleggtom@curii.com
<ul></ul><p>LGTM, thanks. The table in <a href="#note-9">#note-9</a> is very helpful / reassuring!</p> Arvados - Bug #19937: build-packages is failing because arvados-cwl-runner is assigned version 2.5.0 instead of a 2.6 dev versionhttps://dev.arvados.org/issues/19937?journal_id=1133272023-03-16T16:25:36ZLucas Di Pentimalucas.dipentima@curii.com
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Resolved</i></li></ul><p>Applied in changeset <a class="changeset" title="Merge branch '19937-build-packages-fix'. Closes #19937 Arvados-DCO-1.1-Signed-off-by: Lucas Di P..." href="https://dev.arvados.org/projects/arvados/repository/arvados/revisions/0e60fbb5966fb7ec8518c4a06d80946c48d9222a">arvados|0e60fbb5966fb7ec8518c4a06d80946c48d9222a</a>.</p>