Skip to content

Fix converter, formatter, logging and import errors - #54

Merged
wolph merged 9 commits into
developfrom
fix/converters-formatters-logging
Oct 2, 2026
Merged

wolph merged 9 commits into
developfrom
fix/converters-formatters-logging

Conversation

@wolph

@wolph wolph commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Found by an adversarial pass over the modules that are unchanged since 4.0.1. Every fix started as a failing test. Only errors whose fix cannot break working code are in here. The rest is listed at the bottom for a decision.

The branch is based on #53, so the test hooks pass on any machine. Merge #53 first, or merge this and #53 is included.

Logging

class Spam(Logged):
    def eggs(self):
        try:
            1 / 0
        except ZeroDivisionError:
            self.exception('boom')    # no traceback in the log
        self.info('hello')            # record says logger.py:info

class Bacon(Logurud): ...
Bacon().info('payload {"a": 1}')      # KeyError: '"a"'

class LoggedInt(Logged, int): ...
LoggedInt(5)                          # 0
  • exception() logs the traceback.
  • Records name the caller instead of logger.py and the wrapper.
  • Logurud accepts a message with a brace in it. An explicit extra still reaches the loguru record.
  • __new__ forwards its arguments, so LoggedInt(5) is 5. A base __new__ that does not take them is called without, as before.

Converters

  • to_int('abc123', regexp=r'\d+') and the to_float equivalent raised IndexError for a pattern without a group. They use the whole match now.
  • scale_1024(0.0001, 9) returned the power -1. It is clamped at 0.

Formatters

Call Before After
timesince(timedelta(days=30)) 1 month and 2 days ago 1 month ago
timesince(timedelta(days=365)) 1 year and 1 day ago 1 year ago
timesince(timedelta(seconds=-1)) 12 months and 4 weeks ago 1 second ago
timesince(aware_datetime) TypeError works
camel_to_underscore('HTTP_OK') htt_p_ok http_ok
camel_to_underscore('HTML5Parser') htm_l5_parser html5_parser

Import

import_global('email.mime.text') raised ImportError unless other code had already imported that submodule. A nested module that exists and fails to import, for example on a missing dependency, shows its own error.

Docstrings corrected, no behaviour change

  • to_float uses the first group, and its default is the int 0.
  • to_str returns bytes. Its examples showed str and only passed because of ALLOW_BYTES.
  • regexp=True drops a sign, a leading decimal point and an exponent.
  • remap is only exact when every value with a fraction is a Decimal.
  • import_global takes a list of names for modules.

Compatibility

What working code can notice:

  • Code that passes stacklevel=2 to a Logged method to work around the wrong caller now points one frame too high.
  • A loguru record no longer carries exc_info, stack_info and stacklevel in its extra.
  • timesince and camel_to_underscore return a different string for the inputs in the table.
  • Logged.exception() outside an except block logs NoneType: None, as logging does.
  • A Logurud message without arguments is logged as written. Code that doubled its braces to avoid the old crash now sees the doubled braces.
  • to_int and to_float with a group-less pattern and an exception argument that caught the old IndexError return the matched value where they returned the default.

A second adversarial pass compared this branch with 4.0.1 and found three regressions in its first version. All three are fixed and pinned by tests:

  • The logger of a Logged class is created at its first instance, as in 4.0.1. Creating it when the class is defined let dictConfig with its default disable_existing_loggers switch it off.
  • A subclass inherits a logger from the class body of its parent.
  • A base class whose __new__ takes no arguments still works next to Logged.

The same pass found nothing that went from right to wrong across 45150 to_int and to_float combinations, 213008 scale_1024 combinations, every timesince day count from 0 to 2000 and 66853 generated identifiers for camel_to_underscore. The caller in the log record is right in 49 call shapes on CPython 3.10 to 3.14 and PyPy 3.10 and 3.11.

Found and left alone

These change what working code sees, so they need a decision:

  • MyClass.info(...) on a Logged class raises AttributeError before the first instance, and a subclass logs under the name of its parent until then. A lazy logger attribute would fix both.
  • camel_to_underscore('IOError') gives ioerror while MyIOError gives my_io_error.
  • import_global(..., modules=[]) imports everything.
  • apply_recursive does not recurse into a dict inside a list and never passes its **kwargs on.
  • python_utils.__all__ lacks containers and exceptions, and logger.__all__ lacks LoggerBase.
  • get_terminal_size returns width 0 for COLUMNS=1.
  • from python_utils.types import * fails on Python 3.15.

wolph added 5 commits October 2, 2026 13:48
The timeout tests counted items against real sleeps and left 10 to 40 ms
of slack. A sleep only promises to take at least as long as requested, so
the counts changed on a busy machine and on a coarse clock:

- Blocking sleeps that overshoot by 40 ms or more made timeout_generator
  yield one item fewer, in five test cases and in its doctest.
- A 15.6 ms event loop clock resolution, the Windows default, let the
  0.05 s timeout fire together with a 0.04 s sleep, so the detector tests
  stopped at 3 instead of 4.

The sync tests and the doctest now run on a fake clock that only moves
when it is slept on, and they check the requested sleeps as well. The
total timeout tests advance the same clock. The per-item timeout tests
yield without waiting and then stall for 10 s against a 0.05 s timeout.
One test stays on the real clock and only checks what holds for any sleep
accuracy.

The fixtures are loaded from a conftest.py in the repository root so the
doctests can use them, and the sdist ships that file.
- exception() logs the traceback. It forwarded exc_info=None, which
  overrode the default of logging.Logger.exception.
- Records name the caller of the log method. They all reported logger.py
  and the name of the wrapper, because the stacklevel was not raised for
  the extra frame. Python 3.10 needs one frame more for exception().
- Logurud no longer crashes on a message with a brace in it. The logging
  keyword arguments were forwarded to loguru, which then ran str.format
  over the message. An explicit extra is still forwarded.
- __new__ forwards its arguments to the next __new__ unless that is
  object.__new__, so a class that combines Logged or Logurud with int or
  str keeps its value.
- A subclass of Logged gets its logger when the class is created. The log
  methods are classmethods and failed before the first instance, and a
  subclass logged under the name of its parent until then.
- to_int and to_float use the whole match for a regexp without a capture
  group. Both raised IndexError, which is not in the default exceptions.
- scale_1024 never returns a negative power. A number below 2**-10 gave
  -1, which indexes a list of prefixes from the wrong end.

The docstrings now say what the code does: to_float uses the first group
and returns the int 0 by default, to_str returns bytes, regexp=True drops
a sign, a leading decimal point and an exponent, and remap is only exact
when every value with a fraction is a Decimal.
- timesince counts every day once. Weeks and days were taken from the
  total number of days, so 30 days was "1 month and 2 days ago".
- timesince describes the size of a negative timedelta. One second in the
  future was "12 months and 4 weeks ago".
- timesince accepts a timezone-aware datetime.
- camel_to_underscore keeps an acronym whole before an underscore or a
  digit. HTTP_OK became htt_p_ok.
A submodule only becomes an attribute of its parent once something has
imported it, so import_global('a.b.c') raised ImportError unless other
code had imported a.b.c first. A missing module still raises the same
ImportError.

The docstring describes the modules argument as the code reads it.
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

Comment thread conftest.py
root because the doctests in ``python_utils`` need them as well.
"""

pytest_plugins: tuple[str, ...] = ('_python_utils_tests.clock',)
wolph added 4 commits October 2, 2026 15:11
test_aio_timeout_generator still counted items against real sleeps. The
case with five sleeps of 0.06 s against a 0.3 s timeout ends one item
short as soon as the sleeps run 15 ms late in total. It failed 3 of 25
runs on a busy machine, and fails every time when asyncio.sleep is made
20 ms late.

The test now lets asyncio.sleep advance the fake clock. The default
iterable test in test_lazy_imports uses the fake clock too, so its 0.05 s
timeout cannot end the loop before the second item.
An adversarial pass compared the fixes on this branch with 4.0.1 and
found behaviour that had changed without being a fix.

- Logged no longer creates its logger when the class is defined. An
  application that configures logging after its imports, with dictConfig
  or fileConfig and their default disable_existing_loggers, switched that
  logger off and lost every line of the class. The logger is created at
  the first instance again. A subclass also inherits a logger from the
  class body of its parent again.
- Logged and Logurud call the next __new__ without arguments when it does
  not take them. Forwarding them broke a base class such as a singleton
  with `def __new__(cls)` and an __init__ that takes arguments.
- import_global passes on the error of a nested module that exists and
  fails to import, such as a missing dependency. It reported the nested
  module itself as missing. It also asks a module for an attribute once,
  and only imports below a module.
- timesince raises TypeError for a date again.
import_global builds the name of a submodule from the __name__ of its
parent. A module object that is stored under another name than its own
made the import fail for that other name, and the error named it instead
of the module that was asked for. It is the same ImportError as for any
missing module again.

The docstring of the constructor helper says that the fallback call
repeats what the first call did before it raised.
@wolph
wolph merged commit 3958f36 into develop Oct 2, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants