format: XML in UTF-16, and a size range that stops failing on sizes a format cannot write

CHANGELOG.md

@@ -58,6 +58,25 @@ because it turns other people's test suites red.
 
 ### Added
 
+- **XML documents can be written in UTF-16.** Two new settings on `xml`:
+  `encoding`, which takes `utf-8`, `utf-16le` or `utf-16be`, and `bom`, which
+  says whether the file opens with a byte order mark. Both default to what
+  these documents have always been, so a recipe that says nothing gets the same
+  bytes it got before.
+
+  The declaration at the top of the file names the encoding the bytes are
+  really in, so a reader is never told one thing and handed another.
+
+  Two things are worth knowing before you use it. A UTF-16 file stores two
+  bytes for every character, so it always has an even number of them and an odd
+  size is refused - the refusal names the nearest size above and below that it
+  can write. And the smallest XML file grows from 264 B to 532 B, because the
+  declaration, the root element and one whole record all cost twice as much.
+
+  `encoding=utf-16le` and `encoding=utf-16be` need `bom=true`. The XML
+  specification requires a mark on a UTF-16 document, and asking for one
+  without it is refused rather than written.
+
 - **SVG drawings can be any size you ask for.** Two new settings on `svg`:
   `width` and `height`, both a whole number of pixels from 1 to 20000. They
   default to the 800 by 600 these drawings have always been, so a recipe that
@@ -375,6 +394,29 @@ because it turns other people's test suites red.
 
 ### Fixed
 
+- **A size range no longer fails on sizes the format cannot write.** Some
+  formats cannot produce every byte count. A file written in UTF-16 always has
+  an even number of bytes, so half of any range was unreachable, and a picture
+  cannot use the handful of byte counts just above its encoded size, because the
+  smallest padding it can add costs more than that.
+
+  Until now a size drawn onto one of those was refused, and the whole run
+  stopped. Whether that happened depended on the seed and the number of files,
+  so the same recipe worked one day and not the next.
+
+  A range asks for some size between two ends rather than for a number, so a
+  drawn size the format cannot write is now moved to the nearest one it can,
+  inside the range that was asked for. Files that moved carry a `size_moved`
+  note in the manifest, and the run says so once.
+
+  Two things are deliberately unchanged. A range that starts below what the
+  format can produce at all is still refused, naming the smallest size it can
+  write, because that is a recipe worth correcting rather than a run worth
+  quietly filling with identical files. And `--size 1001` still refuses, because
+  that named a number.
+
+  Runs that worked before are byte for byte what they were.
+
 - **The window now warns when a run's record will be too big for this build to
   read back.** The command line has said this since the ceiling was measured.
   The window said nothing at all, so somebody who generated 25 000 files from it

README.md

@@ -476,10 +476,10 @@ recipe. `tfg formats <id>` prints the allowed range or list for each:
 | `pptx` | `slides` |
 | `csv` | `delimiter`, `line_ending`, `header`, `quote_style`, `columns` |
 | `log` | `entry_format`, `timestamps`, `rate`, `methods`, `status_mix`, `level_mix`, `ip_version`, `line_ending` |
-| `txt`, `md` | `encoding`, `bom` |
+| `txt`, `md`, `xml` | `encoding`, `bom` |
 | `json` | `formatting` |
 | `svg` | `width`, `height` |
-| `xml`, `html` | none |
+| `html` | none |
 
 ```
 tfg generate --format jpg --size 500kb --set width=1920 --set height=1080 --set quality=85

internal/engine/drawsizes.go

@@ -0,0 +1,171 @@
+package engine
+
+import (
+	"errors"
+
+	"github.com/donislawdev/TestingFilesGenerator/internal/core"
+	"github.com/donislawdev/TestingFilesGenerator/internal/format"
+)
+
+// Settling the size of every file of a range target.
+//
+// Taken out of engine.go on 2026-09-08, when snapping arrived and that file
+// went past the size ceiling. The line is what a part does rather than how long
+// it is: everything here answers "what size does each file of this range get",
+// and nothing else in the engine asks that question.
+
+// firstWritable is the smallest size at or above from that this format will
+// really write, without going past limit.
+//
+// It JUMPS rather than scans, and that is the whole reason it is cheap enough
+// to run for every file. A format refusing a size it cannot write already names
+// the next one it can, in BelowMinimumError.Minimum, so one refusal is normally
+// one step. Measured 2026-09-08 on the two shapes this project has: an odd size
+// under UTF-16 answers with size+1, and a PNG size inside the band above its
+// encoded picture answers with the top of that band - 143 B answers 154 B.
+//
+// When nothing in the span is writable it hands back the format's OWN refusal
+// for the first size it tried. That refusal already carries the four parts a
+// refusal owes and names a size that would work, so there is no second wording
+// here to drift away from it.
+//
+// The judge is the generator itself rather than a second copy of its rules. A
+// copy would be a place for the two to disagree, and the disagreement would
+// surface as a size that drawing accepted and writing refused.
+//
+// Descriptor.SmallestAccepted is this function's sibling and walks the same
+// jump, from zero and with no ceiling. They are NOT one function, and the reason
+// is the crash guard: everything the engine asks of a generator goes through
+// planWithoutCrashing, so a panic costs one file rather than the process, while
+// SmallestAccepted is called from guards and from the window, where that wrapper
+// is not in hand. Merging them would mean handing a planner in, which buys less
+// than it costs.
+func firstWritable(desc format.Descriptor, r format.Request, from, limit int64) (int64, error) {
+	var first error
+	for at := from; at <= limit; {
+		r.Bytes = at
+		_, err := planWithoutCrashing(desc, r)
+		if err == nil {
+			return at, nil
+		}
+		if first == nil {
+			first = err
+		}
+		var below *format.BelowMinimumError
+		if !errors.As(err, &below) || below.Minimum <= at {
+			// Not a refusal about the size, or one that names no way forward.
+			// Either way there is nothing to jump to.
+			return 0, err
+		}
+		at = below.Minimum
+	}
+	return 0, first
+}
+
+// drawSizes settles the size of every file of a range target.
+//
+// Every size is settled here, before a single byte is written, and that order
+// is the point rather than an optimisation. A tool whose whole promise is that
+// the same seed gives the same run cannot have an error that appears and
+// disappears with the count.
+//
+// A DRAWN SIZE IS SNAPPED to one the format can write, and that is what this
+// function gained on 2026-09-08. Until then it drew from the interval and hoped:
+// a size the format could not write was refused later, by the per file plan, and
+// whether that happened depended on what came out of the seed. Two shapes cause
+// it and neither is rare. Four formats have unreachable BANDS - PNG cannot use
+// the eleven byte counts above a picture's encoded size, because the smallest
+// padding chunk costs twelve, and the OPC three declare the same shape. Three
+// more have unreachable PARITY - a UTF-16 file is a whole number of sixteen bit
+// units, so half of every range is unwritable, which took `--size-range
+// 1000-1010` down about half the time (O190).
+//
+// Snapping is not rounding, and rule 1 is untouched. A range is a request for
+// SOME size between two ends, not for a number - so answering with a writable
+// size inside it is the answer, while `--size 1001` still refuses because that
+// one named a number.
+//
+// PER FILE, not once for the target, because writability moves with the SEED as
+// well as with the size: the same 64x64 PNG recipe has a floor of 144, 143 and
+// 144 B at seeds 1, 2 and 3. Judging file 0 says nothing about file 2, and this
+// comment claimed otherwise until 2026-09-08.
+//
+// What was written here before, and is now obsolete: closing this "needs the
+// format to declare its unreachable bands, which is a change to
+// format.Descriptor and the owner's call". It needed no such thing. The refusal
+// ALREADY carries the next writable size, so probing and jumping does it with
+// no new surface - measured before the change rather than argued.
+//
+// Bytes do not move for any run that worked before. A run that succeeds today
+// has every drawn size writable, or it would have failed, and snapping a
+// writable size returns it unchanged.
+func drawSizes(t *Target, desc format.Descriptor, targetSeed uint64) error {
+	first := format.Request{
+		Contains:   t.Contains,
+		Seed:       core.FileSeed(targetSeed, 0),
+		Label:      t.Label,
+		Properties: t.Properties,
+	}
+
+	// The low end is judged against what this format can do AT ALL, and that
+	// check is older than the snapping below it. The two answer different
+	// questions and both are wanted.
+	//
+	// A range starting under the format's floor is a recipe somebody should
+	// fix: asking PDF for 10 B to 8 kB says a spread was wanted and most of it
+	// does not exist, so the honest answer is the format's own refusal naming
+	// its floor, not forty files quietly piled on it. A range starting at or
+	// above the floor whose SOME sizes are unwritable - odd numbers under
+	// UTF-16, the band above a PNG's encoded picture - is a different thing:
+	// nobody can be expected to enumerate those, and snapping inside the range
+	// is the answer.
+	floor, err := firstWritable(desc, first, 0, t.SizeMax)
+	if err != nil {
+		// The floor is above the whole range, so nothing in it is writable.
+		return err
+	}
+	if t.SizeMin < floor {
+		// Asked again at the low end so the format words its own refusal, with
+		// the number the person actually wrote.
+		first.Bytes = t.SizeMin
+		if _, err := planWithoutCrashing(desc, first); err != nil {
+			return err
+		}
+	}
+
+	span := uint64(t.SizeMax - t.SizeMin)
+	t.SizeMoved = make([]bool, len(t.Sizes))
+
+	for i := range t.Sizes {
+		want := t.SizeMin
+		if span != 0 {
+			// Per index, never from a running stream. Raising a count then
+			// leaves the sizes of the earlier files alone, which is rule 2 and
+			// the reason core.SizeSeed takes an index at all.
+			r := core.NewRand(core.SizeSeed(targetSeed, i))
+			want = t.SizeMin + int64(r.Uint64N(span+1))
+		}
+
+		req := format.Request{
+			Contains:   t.Contains,
+			Seed:       core.FileSeed(targetSeed, i),
+			Label:      t.Label,
+			Properties: t.Properties,
+		}
+
+		got, err := firstWritable(desc, req, want, t.SizeMax)
+		if err != nil && want > t.SizeMin {
+			// Nothing writable from the draw upwards. The bottom of the range
+			// can still hold something - a draw landing on the last odd number
+			// of a range has nowhere above it and plenty below - so the range is
+			// only empty once THAT fails too.
+			got, err = firstWritable(desc, req, t.SizeMin, t.SizeMax)
+		}
+		if err != nil {
+			return err
+		}
+		t.SizeMoved[i] = got != want
+		t.Sizes[i] = got
+	}
+	return nil
+}

internal/engine/engine.go

@@ -51,6 +51,15 @@ type Target struct {
 	SizeIsRange bool
 	SizeMin     int64
 	SizeMax     int64
+	// SizeMoved marks the files whose drawn size was not one the format can
+	// write, so the nearest writable one was used instead. Empty for a target
+	// that is not a range.
+	//
+	// Kept so the move can be REPORTED rather than done quietly. Silence is
+	// banned, and a person who asked for a spread and got one size back should
+	// not have to find that out by reading a manifest they had no reason to
+	// open.
+	SizeMoved []bool
 	// BoundaryLimit is the limit a boundary set was built around, zero when
 	// this target is not one. The three files name themselves from it.
 	//
@@ -70,71 +79,6 @@ type Target struct {
 	Properties map[string]string
 }
 
-// drawSizes settles the size of every file of a range target.
-//
-// Judged at the low end before a single size is drawn, and that order is the
-// point rather than an optimisation. A range whose low end the format cannot
-// deliver - below the minimum of the format, or too small to hold what the
-// container was told to hold - would otherwise fail on some runs and not
-// others, depending on what came out of the seed. A tool whose whole promise
-// is that the same seed gives the same run cannot have an error that appears
-// and disappears.
-//
-// THE LOW END IS NOT THE WHOLE ANSWER, and this comment claimed it was
-// until 2026-09-06. It said the range "either works for every file or for
-// none", and the code does not provide that. The check here is sufficient only
-// if a format's reachable sizes are one unbroken interval starting at its
-// minimum, and for four of them they are not: PNG has an unreachable band of
-// eleven byte counts immediately above every picture's encoded size, because
-// the smallest padding chunk costs twelve bytes, and the OPC three declare the
-// same shape between the comment capacity and the smallest extra part.
-//
-// So a size DRAWN into such a band is refused later, by the per file plan, and
-// whether that happens depends on the count. Measured on 2026-09-06, one 64x64
-// PNG recipe at one seed with size-range 143-200: counts 1 and 2 are accepted,
-// counts 3, 5, 8, 12, 20 and 40 are refused. The low end moves with the seed
-// too - 144, 143, 144 B at seeds 1, 2 and 3 - so judging file 0's band says
-// nothing about file 2's.
-//
-// The bytes are stable under a raised count and that was verified, so rule 2
-// holds for CONTENT. What is not stable is whether the run happens at all.
-// Closing that needs the format to declare its unreachable bands so the whole
-// interval can be judged before anything is drawn, which is a change to
-// format.Descriptor and the owner's call. Until then the refusal at least
-// names the key the recipe carries - see atTarget - rather than pointing at a
-// "size" setting a range target does not have.
-//
-// The judge is the generator itself rather than a second copy of its rules
-// here. A copy would be a place for the two to disagree, and the disagreement
-// would surface as a file that planning accepted and writing refused.
-func drawSizes(t *Target, desc format.Descriptor, targetSeed uint64) error {
-	if _, err := planWithoutCrashing(desc, format.Request{
-		Bytes:      t.SizeMin,
-		Contains:   t.Contains,
-		Seed:       core.FileSeed(targetSeed, 0),
-		Label:      t.Label,
-		Properties: t.Properties,
-	}); err != nil {
-		return err
-	}
-
-	span := uint64(t.SizeMax - t.SizeMin)
-	for i := range t.Sizes {
-		if span == 0 {
-			// Both ends the same is legal and means identical files. Drawing
-			// from a range of one is not wrong, it just reads worse.
-			t.Sizes[i] = t.SizeMin
-			continue
-		}
-		// Per index, never from a running stream. Raising a count then leaves
-		// the sizes of the earlier files alone, which is rule 2 and the reason
-		// core.SizeSeed takes an index at all.
-		r := core.NewRand(core.SizeSeed(targetSeed, i))
-		t.Sizes[i] = t.SizeMin + int64(r.Uint64N(span+1))
-	}
-	return nil
-}
-
 // Uniform is n files of the same size, which is what most targets ask for.
 func Uniform(n int, bytes int64) []int64 {
 	if n <= 0 {

internal/engine/plantarget.go

@@ -47,6 +47,21 @@ func (pl *planning) files(ctx context.Context, t *Target, desc format.Descriptor
 			return atTarget(position, t, err)
 		}
 
+		// A size drawn from the range that this format cannot write was moved
+		// to the nearest one it can. Silence is banned, so it is said here.
+		//
+		// The wording carries NO number on purpose. Notes are grouped by their
+		// text, so a number would make every file its own line - 25 000 of them
+		// on a big run, which is what buried the one note that mattered before
+		// grouping arrived. Which files moved is still exact in the manifest,
+		// because the note sits on each of their entries.
+		if idx < len(t.SizeMoved) && t.SizeMoved[idx] {
+			p.Notes = append(p.Notes, format.Note{
+				Code:   "size_moved",
+				Detail: "A size drawn from the range is not one this format can write, so the nearest size it can write was used instead. The file is still inside the range that was asked for.",
+			})
+		}
+
 		name, err := renderName(t, desc, idx)
 		if err != nil {
 			return atTarget(position, t, err)