Project

General

Profile

Actions

Story #18797

closed

Flesh out python sdk documentation in docstrings & ensure good presentation in pydoc

Added by Peter Amstutz 12 months ago. Updated about 2 months ago.

Status:
Resolved
Priority:
Normal
Assigned To:
Category:
SDKs
Target version:
Start date:
11/11/2022
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Story points:
-
Release relationship:
Auto

Description

Determine the capabilities & correct usage of pydoc for
  • text formatting
  • annotating function parameters
  • module-level docstrings
  • treatment of Py3 type annotations (what does the documentation system do with them, how are they displayed, how does it take advantage of it)

Determine how to mark things as deprecated (documentation only, or possibly using decorators somehow)

Determine how to hide private variables/functions/classes from documentation (or possibly not export them at all, but need to be careful about backwards compatibility)

Do some initial documentation samples and share with the team for feedback.

The end goal is to update all documentation for public APIs in the Python SDK with correct formatting, deprecation markers, type annotations, etc. Also module-level docstrings to direct users to the correct classes for various topics. However, documenting everything now is not required for this ticket.

You should produce a document with guidelines and best practices for the team for Python SDK documentation going forward.


Files

arvados.retry.txt (7.85 KB) arvados.retry.txt pydoc output Brett Smith, 11/10/2022 10:06 PM
Screenshot from 2022-11-10 16-48-43.png (193 KB) Screenshot from 2022-11-10 16-48-43.png pdoc method output Brett Smith, 11/10/2022 10:06 PM
Screenshot from 2022-11-10 16-54-35.png (220 KB) Screenshot from 2022-11-10 16-54-35.png pdoc class output Brett Smith, 11/10/2022 10:06 PM

Subtasks 3 (0 open3 closed)

Task #19725: group reviewResolved11/11/2022

Actions
Task #19788: Review 18797-retry-docstringsResolvedLucas Di Pentima11/21/2022

Actions
Task #19789: Review Coding_Standards#PythonResolvedLucas Di Pentima11/21/2022

Actions

Related issues

Related to Arvados - Support #18263: Plan to document the Python SDKResolvedPeter Amstutz

Actions
Related to Arvados Epics - Story #18800: Update Python SDK documentationNew11/01/202204/30/2023

Actions
Actions #1

Updated by Peter Amstutz 12 months ago

  • Status changed from New to In Progress
Actions #2

Updated by Peter Amstutz 12 months ago

  • Description updated (diff)
Actions #3

Updated by Peter Amstutz 12 months ago

  • Description updated (diff)
Actions #4

Updated by Peter Amstutz 12 months ago

  • Description updated (diff)
Actions #5

Updated by Peter Amstutz 12 months ago

  • Description updated (diff)
Actions #6

Updated by Peter Amstutz 12 months ago

Actions #7

Updated by Peter Amstutz 12 months ago

  • Related to Story #18800: Update Python SDK documentation added
Actions #8

Updated by Peter Amstutz 3 months ago

  • Target version set to 2022-11-23 sprint
Actions #9

Updated by Peter Amstutz 3 months ago

  • Assigned To set to Brett Smith
Actions #10

Updated by Peter Amstutz 3 months ago

  • Description updated (diff)
Actions #11

Updated by Brett Smith 3 months ago

Some initial big-picture thoughts.

Determine the capabilities & correct usage of pydoc for
  • text formatting

The pydoc CLI tool doesn't do any high-level text formatting like man does. The text in the docstring is the text it renders. Because of that, the way to get the best rendering across both pydoc and HTML documentation is to pick a plain text markup that looks good either way.

There are a few options here but I think our realistic choice is reStructuredText vs. Markdown. reST is the best-supported format for Python documentation, and it does have more directive for higher-level formatting. Markdown is of course more popular generally and we're already using it so it's one less thing for the team to learn.

For us I think I would vote Markdown, with a style guide that emphasizes plaintext readability (e.g., footnote-style links, underlined headers, etc.). But either is fine.

  • annotating function parameters

The only officially supported annotations are type annotations, see below. Beyond that, there are conventions for documenting arguments in docstrings, but they're just that, conventions. Not even the official style guide PEP 257 has a suggestion. We should just settle on one and it should become part of the document.

  • module-level docstrings

PEP 257 does have thoughts about this. I quote:

The docstring for a module should generally list the classes, exceptions and functions (and any other objects) that are exported by the module, with a one-line summary of each. (These summaries generally give less detail than the summary line in the object’s docstring.) The docstring for a package (i.e., the docstring of the package’s __init__.py module) should also list the modules and subpackages exported by the package.

  • treatment of Py3 type annotations (what does the documentation system do with them, how are they displayed, how does it take advantage of it)

pydoc will display them basically the same as they're written in the source code. So do most modern documentation publishing systems, including the one we currently use, pdoc3.

Determine how to mark things as deprecated (documentation only, or possibly using decorators somehow)

It should both be documented and raise a DeprecationWarning. The documentation will ultimately be our own convention again, it's just another thing to pick and be consistent about.

DeprecationWarnings are done with the standard warnings module . These warnings can be filtered, redirected, etc. by users, so it has all the features you'd want from an SDK like this.

With some care you could maybe write a decorator to handle both these things. The main thing is to make sure that all this stuff tells people what they should use instead.

Determine how to hide private variables/functions/classes from documentation (or possibly not export them at all, but need to be careful about backwards compatibility)

This isn't doable as such. The best you can do is follow the convention of naming these things with a leading underscore, and then reinforce in the documentation you mean it.

pdoc3 will let you configure it to hide documentation for some of these things, but that's specific to pdoc3, other tools like pydoc won't respect it.

Actions #12

Updated by Brett Smith 3 months ago

I am currently leaning toward a style that starts with PEP 257 and then says:

  • Format docstrings with Markdown.
  • Document function arguments in definition lists. Ideally include a type in the defined term.
  • Document deprecated functions with the admonition `!!! deprecated` right after the summary line. Make sure to document what people should use instead. We should consider if there are other types of admonitions we want to standardize.
  • Quote identifiers with backticks so that pdoc can link them. Use fully qualified names for identifiers outside the scope of the current module.
  • Use footnote style for links.
  • Use underline style for headers.

I've tried out how this would look on arvados.retry since that's relatively small and self-contained and originally written by me. I've pushed 18797-docstring-style-wip with my changes to the branch. I've attached how it ends up looking in pdoc and pydoc. In the screenshots, note the code formatting with syntax highlighting, identifier links, and Deprecated callout.

Actions #13

Updated by Lucas Di Pentima 3 months ago

18797-retry-docstrings looks good to merge, thanks!

Actions #14

Updated by Lucas Di Pentima 3 months ago

Wiki updates on Coding Standards are clear and to the point, thanks.

Actions #15

Updated by Brett Smith 3 months ago

  • Status changed from In Progress to Resolved
Actions #16

Updated by Peter Amstutz about 2 months ago

  • Release set to 47
Actions

Also available in: Atom PDF