Feature #17013

Release python wheels

Added by Peter Amstutz 3 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
SDKs
Target version:
Start date:
11/23/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
1.0

Description

https://github.com/arvados/arvados/pull/73

Related, I noticed this code in run_upload_packages.py

        # NOTE: If we ever start uploading Python 3 packages, we'll need to
        # figure out some way to adapt cmd to match.  It might be easiest
        # to give all our setup.py files the executable bit, and run that
        # directly.
        # We also must run `sdist` before `upload`: `upload` uploads any
        # distributions previously generated in the command.  It doesn't
        # know how to upload distributions already on disk.  We write the
        # result to a dedicated directory to avoid interfering with our
        # timestamp tracking.
        cmd = ['python2.7', 'setup.py']

Subtasks

Task #17144: Review 17013-upload-wheels-to-pypi in the arvados-dev repoResolvedWard Vandewege


Related issues

Related to Arvados - Bug #17159: [build] make build-packages-python-ruby run on a Jenkins satelliteNew

Related to Arvados - Bug #17160: [packaging] upgrade run_upload_packages.py to use twine for upload to PyPINew

Associated revisions

Revision 20ffc42d (diff)
Added by Ward Vandewege about 2 months ago

We use Python3 now, exclusively.

refs #17013

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

Revision e1c58d57
Added by Ward Vandewege about 2 months ago

Merge branch '17013-upload-wheels-to-pypi' into master

closes #17013

Arvados-DCO-1.1-Signed-off-by: Ward Vandewege <>

History

#1 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#2 Updated by Peter Amstutz 3 months ago

  • Description updated (diff)

#3 Updated by Peter Amstutz 3 months ago

  • Category set to SDKs

#4 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-11-04 Sprint to 2020-11-18

#5 Updated by Peter Amstutz 3 months ago

  • Target version changed from 2020-11-18 to 2020-12-02 Sprint

#6 Updated by Peter Amstutz 2 months ago

  • Story points set to 1.0

#7 Updated by Peter Amstutz 2 months ago

  • Assigned To set to Ward Vandewege

#8 Updated by Ward Vandewege about 2 months ago

  • Status changed from New to In Progress
        # NOTE: If we ever start uploading Python 3 packages, we'll need to
        # figure out some way to adapt cmd to match.  It might be easiest
        # to give all our setup.py files the executable bit, and run that
        # directly.
        # We also must run `sdist` before `upload`: `upload` uploads any
        # distributions previously generated in the command.  It doesn't
        # know how to upload distributions already on disk.  We write the
        # result to a dedicated directory to avoid interfering with our
        # timestamp tracking.
        cmd = ['python2.7', 'setup.py']

I've fixed that to use python3.

#9 Updated by Ward Vandewege about 2 months ago

Ready for review in commit:c44efafc9e35941d0ebf1fdc819cd848024b82ff on branch 17013-upload-wheels-to-pypi in the arvados-dev repository.

This doesn't upgrade to using 'twine' for upload, which is now apparently preferred (cf. https://packaging.python.org/guides/distributing-packages-using-setuptools/?highlight=--dist-dir#upload-your-distributions). We don't have a recent version of twine on our build server yet.

When this is merged I intend to merge PR #73 (https://github.com/arvados/arvados/pull/73) to make the wheels universal for the Arvados Python SDK. I will see about making the same change for arvados-cwl-runner, and if possible, arvados-fuse (if llfuse doesn't get in the way).

#10 Updated by Peter Amstutz about 2 months ago

Ward Vandewege wrote:

Ready for review in commit:c44efafc9e35941d0ebf1fdc819cd848024b82ff on branch 17013-upload-wheels-to-pypi in the arvados-dev repository.

This doesn't upgrade to using 'twine' for upload, which is now apparently preferred (cf. https://packaging.python.org/guides/distributing-packages-using-setuptools/?highlight=--dist-dir#upload-your-distributions). We don't have a recent version of twine on our build server yet.

All the documentation says to use twine, so we will definitely want to do that. But if "setup.py upload" still works that's fine for now.

When this is merged I intend to merge PR #73 (https://github.com/arvados/arvados/pull/73) to make the wheels universal for the Arvados Python SDK. I will see about making the same change for arvados-cwl-runner, and if possible, arvados-fuse (if llfuse doesn't get in the way).

I don't think we actually want to make universal wheels? That means they work on both Python 2 and Python 3, and we don't support Python 2 any more. We want pure-python py3 wheels.

#11 Updated by Ward Vandewege about 2 months ago

  • % Done changed from 0 to 100
  • Status changed from In Progress to Resolved

#12 Updated by Ward Vandewege about 2 months ago

  • Related to Bug #17159: [build] make build-packages-python-ruby run on a Jenkins satellite added

#13 Updated by Ward Vandewege about 2 months ago

  • Related to Bug #17160: [packaging] upgrade run_upload_packages.py to use twine for upload to PyPI added

#14 Updated by Ward Vandewege about 2 months ago

Peter Amstutz wrote:

This doesn't upgrade to using 'twine' for upload, which is now apparently preferred (cf. https://packaging.python.org/guides/distributing-packages-using-setuptools/?highlight=--dist-dir#upload-your-distributions). We don't have a recent version of twine on our build server yet.

All the documentation says to use twine, so we will definitely want to do that. But if "setup.py upload" still works that's fine for now.

Yes, "setup.py upload" still works and it has worked for many years. twine just seems like the latest fad. I'm sure there will be another tool that will be "strongly recommended" in a year or two, while all the other ways keep working. Pfff.

We should make this CI job run in a satellite instance that is more up to date, and then we can switch to using twine. I've made a few tickets for that work (#17159 and #17160).

When this is merged I intend to merge PR #73 (https://github.com/arvados/arvados/pull/73) to make the wheels universal for the Arvados Python SDK. I will see about making the same change for arvados-cwl-runner, and if possible, arvados-fuse (if llfuse doesn't get in the way).

I don't think we actually want to make universal wheels? That means they work on both Python 2 and Python 3, and we don't support Python 2 any more. We want pure-python py3 wheels.

Fair enough. This patch builds "pure" wheels, so then we are all set.

Also available in: Atom PDF