Fix converter, formatter, logging and import errors - #54
Merged
Merged
Conversation
The timeout tests counted items against real sleeps and left 10 to 40 ms of slack. A sleep only promises to take at least as long as requested, so the counts changed on a busy machine and on a coarse clock: - Blocking sleeps that overshoot by 40 ms or more made timeout_generator yield one item fewer, in five test cases and in its doctest. - A 15.6 ms event loop clock resolution, the Windows default, let the 0.05 s timeout fire together with a 0.04 s sleep, so the detector tests stopped at 3 instead of 4. The sync tests and the doctest now run on a fake clock that only moves when it is slept on, and they check the requested sleeps as well. The total timeout tests advance the same clock. The per-item timeout tests yield without waiting and then stall for 10 s against a 0.05 s timeout. One test stays on the real clock and only checks what holds for any sleep accuracy. The fixtures are loaded from a conftest.py in the repository root so the doctests can use them, and the sdist ships that file.
- exception() logs the traceback. It forwarded exc_info=None, which overrode the default of logging.Logger.exception. - Records name the caller of the log method. They all reported logger.py and the name of the wrapper, because the stacklevel was not raised for the extra frame. Python 3.10 needs one frame more for exception(). - Logurud no longer crashes on a message with a brace in it. The logging keyword arguments were forwarded to loguru, which then ran str.format over the message. An explicit extra is still forwarded. - __new__ forwards its arguments to the next __new__ unless that is object.__new__, so a class that combines Logged or Logurud with int or str keeps its value. - A subclass of Logged gets its logger when the class is created. The log methods are classmethods and failed before the first instance, and a subclass logged under the name of its parent until then.
- to_int and to_float use the whole match for a regexp without a capture group. Both raised IndexError, which is not in the default exceptions. - scale_1024 never returns a negative power. A number below 2**-10 gave -1, which indexes a list of prefixes from the wrong end. The docstrings now say what the code does: to_float uses the first group and returns the int 0 by default, to_str returns bytes, regexp=True drops a sign, a leading decimal point and an exponent, and remap is only exact when every value with a fraction is a Decimal.
- timesince counts every day once. Weeks and days were taken from the total number of days, so 30 days was "1 month and 2 days ago". - timesince describes the size of a negative timedelta. One second in the future was "12 months and 4 weeks ago". - timesince accepts a timezone-aware datetime. - camel_to_underscore keeps an acronym whole before an underscore or a digit. HTTP_OK became htt_p_ok.
A submodule only becomes an attribute of its parent once something has
imported it, so import_global('a.b.c') raised ImportError unless other
code had imported a.b.c first. A missing module still raises the same
ImportError.
The docstring describes the modules argument as the code reads it.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
| root because the doctests in ``python_utils`` need them as well. | ||
| """ | ||
|
|
||
| pytest_plugins: tuple[str, ...] = ('_python_utils_tests.clock',) |
test_aio_timeout_generator still counted items against real sleeps. The case with five sleeps of 0.06 s against a 0.3 s timeout ends one item short as soon as the sleeps run 15 ms late in total. It failed 3 of 25 runs on a busy machine, and fails every time when asyncio.sleep is made 20 ms late. The test now lets asyncio.sleep advance the fake clock. The default iterable test in test_lazy_imports uses the fake clock too, so its 0.05 s timeout cannot end the loop before the second item.
… fix/converters-formatters-logging
An adversarial pass compared the fixes on this branch with 4.0.1 and found behaviour that had changed without being a fix. - Logged no longer creates its logger when the class is defined. An application that configures logging after its imports, with dictConfig or fileConfig and their default disable_existing_loggers, switched that logger off and lost every line of the class. The logger is created at the first instance again. A subclass also inherits a logger from the class body of its parent again. - Logged and Logurud call the next __new__ without arguments when it does not take them. Forwarding them broke a base class such as a singleton with `def __new__(cls)` and an __init__ that takes arguments. - import_global passes on the error of a nested module that exists and fails to import, such as a missing dependency. It reported the nested module itself as missing. It also asks a module for an attribute once, and only imports below a module. - timesince raises TypeError for a date again.
import_global builds the name of a submodule from the __name__ of its parent. A module object that is stored under another name than its own made the import fail for that other name, and the error named it instead of the module that was asked for. It is the same ImportError as for any missing module again. The docstring of the constructor helper says that the fallback call repeats what the first call did before it raised.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found by an adversarial pass over the modules that are unchanged since 4.0.1. Every fix started as a failing test. Only errors whose fix cannot break working code are in here. The rest is listed at the bottom for a decision.
The branch is based on #53, so the test hooks pass on any machine. Merge #53 first, or merge this and #53 is included.
Logging
exception()logs the traceback.logger.pyand the wrapper.Logurudaccepts a message with a brace in it. An explicitextrastill reaches the loguru record.__new__forwards its arguments, soLoggedInt(5)is5. A base__new__that does not take them is called without, as before.Converters
to_int('abc123', regexp=r'\d+')and theto_floatequivalent raisedIndexErrorfor a pattern without a group. They use the whole match now.scale_1024(0.0001, 9)returned the power-1. It is clamped at 0.Formatters
timesince(timedelta(days=30))1 month and 2 days ago1 month agotimesince(timedelta(days=365))1 year and 1 day ago1 year agotimesince(timedelta(seconds=-1))12 months and 4 weeks ago1 second agotimesince(aware_datetime)TypeErrorcamel_to_underscore('HTTP_OK')htt_p_okhttp_okcamel_to_underscore('HTML5Parser')htm_l5_parserhtml5_parserImport
import_global('email.mime.text')raisedImportErrorunless other code had already imported that submodule. A nested module that exists and fails to import, for example on a missing dependency, shows its own error.Docstrings corrected, no behaviour change
to_floatuses the first group, and its default is the int0.to_strreturnsbytes. Its examples showedstrand only passed because ofALLOW_BYTES.regexp=Truedrops a sign, a leading decimal point and an exponent.remapis only exact when every value with a fraction is aDecimal.import_globaltakes a list of names formodules.Compatibility
What working code can notice:
stacklevel=2to aLoggedmethod to work around the wrong caller now points one frame too high.exc_info,stack_infoandstacklevelin itsextra.timesinceandcamel_to_underscorereturn a different string for the inputs in the table.Logged.exception()outside anexceptblock logsNoneType: None, asloggingdoes.Logurudmessage without arguments is logged as written. Code that doubled its braces to avoid the old crash now sees the doubled braces.to_intandto_floatwith a group-less pattern and anexceptionargument that caught the oldIndexErrorreturn the matched value where they returned the default.A second adversarial pass compared this branch with 4.0.1 and found three regressions in its first version. All three are fixed and pinned by tests:
Loggedclass is created at its first instance, as in 4.0.1. Creating it when the class is defined letdictConfigwith its defaultdisable_existing_loggersswitch it off.loggerfrom the class body of its parent.__new__takes no arguments still works next toLogged.The same pass found nothing that went from right to wrong across 45150
to_intandto_floatcombinations, 213008scale_1024combinations, everytimesinceday count from 0 to 2000 and 66853 generated identifiers forcamel_to_underscore. The caller in the log record is right in 49 call shapes on CPython 3.10 to 3.14 and PyPy 3.10 and 3.11.Found and left alone
These change what working code sees, so they need a decision:
MyClass.info(...)on aLoggedclass raisesAttributeErrorbefore the first instance, and a subclass logs under the name of its parent until then. A lazyloggerattribute would fix both.camel_to_underscore('IOError')givesioerrorwhileMyIOErrorgivesmy_io_error.import_global(..., modules=[])imports everything.apply_recursivedoes not recurse into a dict inside a list and never passes its**kwargson.python_utils.__all__lackscontainersandexceptions, andlogger.__all__lacksLoggerBase.get_terminal_sizereturns width 0 forCOLUMNS=1.from python_utils.types import *fails on Python 3.15.