Project

General

Profile

Actions

Idea #11419

closed

[SDKs] support text-mode open() in Python 3

Added by Tom Clegg almost 7 years ago. Updated about 5 years ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
-
Target version:
Story points:
-
Release relationship:
Auto

Subtasks 1 (0 open1 closed)

Task #14547: Review 11419-text-modeResolvedTom Clegg03/21/2017Actions

Related issues

Related to Arvados - Idea #11770: [Python SDK] Implement support for universal newline mode in Collections APINew05/26/2017Actions
Related to Arvados - Idea #11308: Support Python3 for arvados-python-client & command line utilitiesResolvedTom Clegg03/21/2017Actions
Blocks Arvados Epics - Idea #14532: [Epic] Port to Python 3 to for Python 2 sunset in December 2019Resolved01/01/202009/16/2020Actions
Actions #1

Updated by Tom Clegg almost 7 years ago

  • Subject changed from open files in text or binary mode to support text-mode open() in Python 3
Actions #2

Updated by Tom Clegg almost 7 years ago

  • Subject changed from support text-mode open() in Python 3 to [SDKs] support text-mode open() in Python 3
  • Assigned To deleted (Tom Clegg)
  • Parent task deleted (#11308)
Actions #3

Updated by Tom Clegg almost 7 years ago

  • Tracker changed from Task to Feature
  • Target version deleted (2017-05-10 sprint)
Actions #4

Updated by Tom Morris over 6 years ago

  • Target version set to Arvados Future Sprints
Actions #5

Updated by Tom Morris over 5 years ago

  • Related to Idea #11308: Support Python3 for arvados-python-client & command line utilities added
Actions #6

Updated by Tom Morris over 5 years ago

  • Target version changed from Arvados Future Sprints to To Be Groomed
Actions #7

Updated by Tom Morris over 5 years ago

  • Release set to 14
Actions #8

Updated by Tom Morris over 5 years ago

  • Tracker changed from Feature to Idea
Actions #9

Updated by Tom Morris over 5 years ago

  • Release deleted (14)
Actions #10

Updated by Tom Morris over 5 years ago

  • Blocks Idea #14532: [Epic] Port to Python 3 to for Python 2 sunset in December 2019 added
Actions #11

Updated by Tom Morris over 5 years ago

  • Target version changed from To Be Groomed to 2018-12-12 Sprint
Actions #12

Updated by Tom Clegg over 5 years ago

  • Assigned To set to Tom Clegg
Actions #13

Updated by Tom Clegg over 5 years ago

11419-text-mode @ 0565661b2c4fb220428ea37c2e0728e4b07d2465 https://ci.curoverse.com/view/Developer/job/developer-run-tests/989/

This change mostly comes down to wrapping the ArvadosFileReader/Writer in an io.BufferedReader/Random and then in an io.TextIOWrapper, if text mode is requested.

With the previous code, the "block boundary isn't on a character boundary" test crashed in Python 2 when using text mode, and the same wrapper fixes it, so I figured it would make sense to fix both at once. However, this does cause a slight API change that could conceivably break existing code. In text mode, the file-like object returned by Collection.open()...
  1. ...is not an ArvadosFileReader/Writer. Code that tests isinstance(f, ...) will behave differently. I found no examples of this in our own repo.
  2. ...cannot seek to arbitrary positions (see TextIOBase seek restrictions). The only occurrence of this I found in our repo was in the "open in text mode in Python 2" test.
  3. ...does not accept close(flush=False). This revealed (via sdk/cwl test failures) that source:sdk/python/arvados/commands/run.py was accidentally using text mode to copy arbitrary files.
Actions #15

Updated by Tom Clegg about 5 years ago

  • Status changed from New to In Progress
Actions #17

Updated by Lucas Di Pentima about 5 years ago

This LGTM, please merge.
Thanks!

Actions #18

Updated by Tom Clegg about 5 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 0 to 100
Actions #19

Updated by Peter Amstutz about 5 years ago

This has a regression:

https://ci.curoverse.com/view/CWL/job/run-cwl-test-4xphq/799/console

2018-12-14 20:28:39 arvados.cwl-runner ERROR: [step-valuefrom5-wf.cwl] While getting final output object: 'TextIOWrapper' object has no attribute 'size'
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/arvados_cwl/runner.py", line 465, in done
    if f.size() > 0:
AttributeError: 'TextIOWrapper' object has no attribute 'size'
Actions #20

Updated by Peter Amstutz about 5 years ago

  • Status changed from Resolved to Feedback
  • Target version changed from 2018-12-12 Sprint to 2018-12-21 Sprint
Actions #21

Updated by Tom Clegg about 5 years ago

11419-cwl-read-json-binary-mode @ fef817dc5d6d1e0160c6c14ba4f41581e241f33a

Actions #22

Updated by Tom Clegg about 5 years ago

  • Status changed from Feedback to Resolved
Actions #23

Updated by Tom Morris about 5 years ago

  • Release set to 15
Actions

Also available in: Atom PDF