Repository navigation
Decide the fate of undocumented script behavior of some modules #93096
Description
There are three dozens of standard modules that can be called via python -m and their documentation doesn't mention it. They can be grouped into five categories:
-
kind of smoke tests:
- codecs:
performs stdin:latin1 → utf-8 → latin1 → stdout passthroughedit: it just wraps stdin and stdout then just exits the script (gh-93096: Removepython -m codecs#94233) - curses.has_key: "Compare the output of this implementation and the ncurses has_key, on platforms where has_key is already available"
- pprint: measures performance (gh-92546: Move pprint benchmark into pyperformance #94613 → Add a benchmark based on
python -m pprintpyperformance#222) - random: evaluates output statistics of supported generators
- codecs:
-
full-fledged crossplatform utils for admin-like users and small automation:
- asyncio: like
pythonbut allows to useawaitin top-level script code - cProfile, profile: runs a script under the profiler
- encodings.rot_13: a stream converter
- filecmp: a crossplatform file comparison utility
- fileinput: prints specified files one by another annotating lines with their source
- http.server: makes a directory available as a site; useful to quickly test a static site with relative links
- mimetypes: useful for batch processing of files (maybe) (gh-93096: Make
mimetypesCLI tool public #93097) - modulefinder: the objdump but for Python source files
- netrc: prints content of
.netrcfor a current user - pdb
- platform: returns a single line like
Windows-10-10.0.19044-SP0; can be useful in automation - quopri: a stream converter
- tabnanny
- wsgiref.simple_server - the same as http.server but for APIs; pases a single request and exits
- asyncio: like
-
both:
- base64:
- a stream converter
-
base64 -tencodes/decodesAladdin:open sesameand tests if the result is the same as the original (gh-93096: Removepython -m base64 -t#94230)
- base64:
-
demos with no real world application:
- curses.textpad: shows an input area; when a user closes it, prints the text back
- ftplib: a simple one-pass FTP downloader (uses ~/.netrc for login)
-
getopt: just passes arguments toThe module is no longer maintained after gh-106535: Soft deprecate the getopt module #105735getopt() - imaplib: sending emails to a dead end has no sence but can be used to check if a email client works or got broken
- shlex: parses stdin using
shlex()and prints the list into stdout - smtplib: a simple e-mail client
- xmlrpc.server: serves a datetime service
-
complex matter; better leave untouched:
- idlelib.*
- tkinter.*
- turtledemo.*
- pstats
Eggs and to-be-removed modules aren't listed.
We need to decide what to do with all these undocumented categories.
I propose the following:
- move smoke tests into
testmodule with deduplication - for full-fledged utils, add
Command-Line Usageinto the docs like in https://docs.python.org/3/library/ast.html#command-line-usage or https://docs.python.org/3/library/trace.html#command-line-usage - move demos into the docs of the corresponding module
Linked PRs
- gh-93096: Remove
-tand-vflags frompickletoolscli #131039 - gh-93096: Remove
-tand-vflags frompicklecli #131068 - gh-93096: Load doctests in
test_pickle#131069 - [3.13] gh-93096: Load doctests in
test_pickle(GH-131069) #131080 - [3.12] gh-93096: Load doctests in
test_pickle(GH-131069) #131081 - gh-93096: Update and document
pickleCLI #131097 - gh-93096: Remove CLI interface for
difflib#131099 - gh-93096: Remove run block in
heapq#131130 - gh-93096: Load doctests in
test_itertools#131133 - [3.13] gh-93096: Load doctests in
test_itertools(GH-131133) #131136 - [3.12] gh-93096: Load doctests in
test_itertools(GH-131133) #131137 - gh-93096: Move
randombenchmark into pyperformance #131144 - gh-93096: Update and document
pickletoolsCLI #131273 - gh-93096: fix
test_mimetypes.test_guess_type_conflicting_with_mimetypes#131408 - gh-93096: fix
test_mimetypes.test_invocation_errortests on iOS ARM64 #132266
Activity
merwok commentedon Mar 10, 2025
I agree on this point – self-tests are better removed.
But the ticket is also about other categories of scripts, which should not be decided by only a few devs here IMO.
pickletools#130973 as a duplicate of this issuegh-93096: Load doctests in `test_pickle` (#131069)
pythongh-93096: Load doctests in `test_pickle` (pythonGH-131069)
pythongh-93096: Load doctests in `test_pickle` (pythonGH-131069)
[3.12] gh-93096: Fix loading doctests in test_pickle (GH-131069) (GH-…
[3.13] gh-93096: Fix loading doctests in test_pickle (GH-131069) (GH-…
gh-93096: Remove `-t` and `-v` flags from `pickletools` cli (#131039)
gh-93096: Remove `-t` and `-v` flags from `pickle` cli (#131068)
donbarbos commentedon Mar 12, 2025
ping @hugovk as author of
--testflag inrandomcliIn discuss you suggested opening a new issue but I think this issue is more appropriate.
Since the discussion was closed I suggest that if anyone has objections, they write here, but I have not seen any objections yet
If you don't mind such a change, then I will open a PR
gh-93096: Load doctests in `test_itertools` (#131133)
pythongh-93096: Load doctests in `test_itertools` (pythonGH-131133)
pythongh-93096: Load doctests in `test_itertools` (pythonGH-131133)
[3.12] gh-93096: Load doctests in `test_itertools` (GH-131133) (#131137)
[3.13] gh-93096: Load doctests in `test_itertools` (GH-131133) (#131136)
donbarbos commentedon Mar 12, 2025
sorry, i didn't wait for you because i found that we already removed such thing for
pprintmodule (PR #94613)these tests are moving to pyperformance, see #131144
gh-93096: Make `mimetypes` CLI tool public (#93097)
gh-93096: Remove CLI interface for `difflib` (#131099)
gh-93096: Update and document `pickle` CLI (#131097)
donbarbos commentedon Mar 16, 2025
We have modules that don't have the
-h/--helpflag, for example:fileinput(no docs, run demo)ftplib(no docs, have help message, but it only prints without any flags)filecmp(no docs)poplib(no docs, run demo)pyclbr(no docs)quopri(no docs)tabnanny(no docs)site(have docs and help message, but it only prints when there is error)The problem is that these CLIs expect some arguments, but nowhere is it said which ones! (no docs, no help messages)
I don't suggest adding documentation for all of them, as this would make them public and this could raise other questions.
I think that users to not have to look into source code, it would be convenient to have a help message and a flag to print it
I don't think help message is required for
asyncio,this,turtledemoor something similar@vstinner does it make sense to open issue to send PRs?
EDIT: I think this will add consistency to the behavior of CLI modules
gh-93096: Update and document `pickletools` CLI (#131273)
pythongh-93096: Make `mimetypes` CLI tool public (python#93097)
pythongh-93096: Remove CLI interface for `difflib` (python#131099)
pythongh-93096: Update and document `pickle` CLI (python#131097)
pythongh-93096: Update and document `pickletools` CLI (python#131273)
gh-93096: fix `test_mimetypes.test_guess_type_conflicting_with_mimety…
gh-93096: fix `test_mimetypes.test_invocation_error` tests on iOS ARM…
pythongh-93096: fix `test_mimetypes.test_invocation_error` tests on i…