Skip to content

fix float: correct rounding for sqrt directed modes, div, nth_root, with_base - #101

Open
cmpute wants to merge 1 commit into
masterfrom
fix/issue-99-100-rounding
Open

cmpute wants to merge 1 commit into
masterfrom
fix/issue-99-100-rounding

Conversation

@cmpute

@cmpute cmpute commented Sep 13, 2026

Copy link
Copy Markdown
Owner

Fixes #99, fixes #100.

Four rounding-correctness bugs, reported as #99 (sqrt) and #100 (div / nth_root / with_base); two more same-class div value bugs were found and fixed along the way. All follow one principle, the same one that fixed Context::mul's operand pre-shrink: never round an operand or an intermediate — carry exact information to one final rounding step.

sqrt (#99)

When the aligned significand's integer root carries precision + 1 digits, the inexact branch hardcoded half-up behavior instead of consulting the rounding mode, so a sticky remainder rounded down under Up/Away — e.g. Up(sqrt(6))² < 6 at precision 53. The rounding now delegates to the mode, with the sticky "at the half" case reported as strictly past the half.

div (#100)

  • The kernel (repr_div) now bounds an over-wide dividend itself, by an exact digit split whose dropped low digits are carried through as sticky rounding information. Context::div's old pre-round of the dividend is gone — it corrupted ties (31/4 base 3 HalfEven gave 6 instead of 9), direction (7/-1 under Up gave -8 instead of -4) and the exactness flag (5/1 @ p1 claimed Exact).
  • The zero-remainder early return is gone: an exact quotient is rounded to the precision in one step (15/3 @ p2 Zero returned the unreduced 3-digit significand 5 instead of Inexact(4)).
  • A precision+1-digit quotient now rounds on the precision grid, not the integer grid — two additional value bugs found while investigating: 3/5 @ p2 returned 0.625 (not even representable; correct is 0.5), 14/3 @ p2 returned 6 instead of 4.

nth_root (#100)

The exponent alignment now takes the truncating shift whenever the padding one would grow the integer root to precision + 1 digits, so the root always carries exactly precision digits and a single rounding decides. The old padding choice rounded the root to an integer first and re-rounded — tying the wrong way at coarse-grid midpoints (nth_root(2, 1.75) @ p2 HalfEven gave 1.0 instead of 1.5).

with_base (#100)

The power-of-base shortcuts and the small-exponent path now round their (possibly wider-than-precision) exact results, and the division path no longer pre-rounds the dividend, so the Exact/Inexact flag is truthful (2.1 → base 2 no longer reports Exact; 3 base4 → base2 @ p1 reduces 0b11).

Performance — no regression (verified with float/benches + callgrind)

  • digit_len on a power-of-two base is now a plain bit length (ilog materialized B^log as a heap buffer only to discard it) — this speeds up every repr_round call site;
  • the dividend-width check runs on integer-only bit-length bounds (the f32-based digits_ub/digits_lb call into libm's log2);
  • the padding branches derive the scaled quotient's digit count instead of re-measuring it;
  • a precision+1-digit quotient's single-step rounding computes its half comparison with at most one multiplication (none in base 2; never for directed modes — the comparator is lazy).
bench (vs master) result
fbig_div 1e1–1e6 parity or faster
dbig_div 1e3/1e4 −11%
dbig_div 1e1 +8% (10-digit operands: the intrinsic cost of the correctly-rounded wide-quotient path that used to mis-round)
nth_root all n ≥ 2 −10…−47% (the truncating alignment also pads less)
exp / ln / trig / hyper −2…−11% (median ≈ −3.5%)

Validation

  • New in-crate regression tests for every reported case (incl. directed-mode bracket sweeps at p = 20/50/100/500, per AGENTS.md);
  • full workspace tests, doc tests, clippy -D warnings, fmt — all green;
  • MPFR differentials in fuzz/: float arithmetic (div tolerance tightened 2 → 1 ulp, since div is now correctly rounded), transcendental incl. directed sqrt/nth_root, trig, complex — all pass.

Note for the issue: only sqrt regressed in 0.6.0 — the div / nth_root / with_base bugs are long-standing (identical in 0.5.2), so OpenDP's 0.5.2 code paths were exposed to them as well.

🤖 Generated with Claude Code

…ith_base

Four rounding-correctness bugs, reported as #99 (sqrt) and #100
(div / nth_root / with_base); two more same-class div value bugs were
found and fixed along the way. All follow one principle, the same one
that fixed Context::mul's operand pre-shrink: never round an operand or
an intermediate — carry exact information to one final rounding step.

- sqrt (#99): when the aligned significand's integer square root carries
  precision+1 digits, the inexact branch hardcoded half-up behavior
  (d*2 >= B → AddOne) instead of consulting the rounding mode, so a
  sticky remainder rounded *down* under Up/Away (Up(sqrt(6))² < 6 at
  p53). It now delegates to R::round_low_part, reporting the sticky
  "at the half" case as strictly past the half.

- div (#100): the kernel bounds an over-wide dividend itself, by an
  exact digit split whose dropped low part (lo) is carried through as
  sticky rounding information; Context::div's old pre-round of the
  dividend is gone (it corrupted ties, direction and the exactness
  flag). The zero-remainder early return is gone too: an exact quotient
  is rounded to the precision in one step, and a precision+1-digit
  quotient now rounds on the precision grid, not the integer grid (3/5
  at p2 returned the unrepresentable 0.625 instead of 0.5). The result
  finally goes through check_finite_exponent on every path.

- nth_root (#100): the exponent alignment now takes the truncating
  shift whenever the padding one would grow the integer root to
  precision+1 digits, so the root always carries exactly precision
  digits and a single rounding decides (the old round-to-integer-then-
  re-round tied the wrong way at coarse-grid midpoints:
  nth_root(2, 1.75) at p2 HalfEven gave 1.0 instead of 1.5).

- with_base (#100): the power-of-base shortcuts and the small-exponent
  path now round their (possibly wider-than-precision) exact results,
  and the division path no longer pre-rounds the dividend, so the
  Exact/Inexact flag is truthful.

No performance regression versus 0.6.0 — verified with float/benches
(primitive div, exp, hyper, trig) at every benched precision, plus
callgrind instruction-count profiles:

- digit_len on a power-of-two base is now a plain bit-length (ilog
  materialized B^log as a heap buffer only to discard it), which
  speeds up every repr_round call site;
- the dividend-width check runs on integer-only bit-length bounds
  (the f32-based digits_ub/digits_lb call into libm's log2);
- the padding branches derive the scaled quotient's digit count
  instead of re-measuring it;
- a precision+1-digit quotient's single-step rounding computes its
  half comparison with at most one multiplication (none in base 2,
  never for directed modes — the comparator is lazy).

FBig/FBig div is at parity or faster everywhere; DBig/DBig div is ~10%
faster at 10^3-10^4 digits and at parity elsewhere except 10-digit
operands (+8%, the intrinsic cost of the correctly-rounded wide-quotient
path that used to mis-round). nth_root is 10-45% faster for every n >= 2
(the truncating alignment also pads less). exp/ln/trig/hyper benches
improve by 2-11% (median ~3.5%) from the allocation-free digit_len.

The fuzz div differential (fuzz/) tightens from 2 ulps to 1 (only the
oracle's own near-tie re-rounding may differ now). Full MPFR
differentials (float arithmetic, transcendental incl. directed sqrt and
nth_root, trig, complex) pass.

Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Div and nth root bugs in rounding or flag Sqrt rounding errors in Dashu 0.6.0

1 participant