From b179e9ee0631f5b58f320ba7a3d8e3e2b378d82e Mon Sep 17 00:00:00 2001 From: DonislawDev Date: Tue, 8 Sep 2026 15:27:16 +0200 Subject: [PATCH] format: XML in UTF-16, and a size range that stops failing on sizes a format cannot write Two changes, and the second was found by the first. XML gains encoding and bom, through the textenc package txt and md already stand on. Both default to what these documents have always been, so the ten pinned hashes are unchanged whether a recipe says nothing or says utf-8 out loud. XML is the first format here whose file SAYS which encoding it is in, so it is the first that can disagree with itself: the declaration follows the bytes, and a guard reads that from the bytes rather than through a decoder, which would have to be told what to expect and would then agree with itself. Measured before any of it was written. Expat refuses a declaration that disagrees with the bytes in both directions, so there is a witness outside this project. It ACCEPTS a UTF-16 document with no byte order mark, though, and reads it correctly - so nothing here could go red on one, and asking for utf-16 without a mark is refused rather than written. Then the range. Some formats cannot write every byte count: a UTF-16 file is always an even number of bytes, and a picture cannot use the handful just above its encoded size. A drawn size landing on one of those took the whole run down, and whether that happened depended on the seed and the count. A range asks for some size between two ends rather than for a number, so a drawn size is now moved to the nearest writable one inside the range, and files that moved carry a note. The comment here said closing this needed formats to declare their unreachable bands, a change to format.Descriptor. It needed no such thing - the refusal already names the next writable size. A range starting below what a format can do at all is still refused, and --size 1001 still refuses, because that one named a number. Ten runs across ten formats come out byte for byte what they were. Co-Authored-By: Claude Opus 5 --- CHANGELOG.md | 42 ++++ README.md | 4 +- internal/engine/drawsizes.go | 171 ++++++++++++++ internal/engine/engine.go | 74 +----- internal/engine/plantarget.go | 15 ++ internal/format/xmlfile/xml.go | 115 +++++++-- internal/guard/layers_test.go | 2 +- internal/guard/parity_test.go | 2 + internal/guard/rangesnap_test.go | 353 ++++++++++++++++++++++++++++ internal/guard/textencoding_test.go | 119 ++++++++-- internal/guard/xmlencoding_test.go | 189 +++++++++++++++ internal/oracle/strict.py | 47 +++- web/public/formats/index.html | 10 + web/public/pl/formaty/index.html | 10 + 14 files changed, 1037 insertions(+), 116 deletions(-) create mode 100644 internal/engine/drawsizes.go create mode 100644 internal/guard/rangesnap_test.go create mode 100644 internal/guard/xmlencoding_test.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 0651326..1cc63b8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -58,6 +58,25 @@ because it turns other people's test suites red. ### Added +- **XML documents can be written in UTF-16.** Two new settings on `xml`: + `encoding`, which takes `utf-8`, `utf-16le` or `utf-16be`, and `bom`, which + says whether the file opens with a byte order mark. Both default to what + these documents have always been, so a recipe that says nothing gets the same + bytes it got before. + + The declaration at the top of the file names the encoding the bytes are + really in, so a reader is never told one thing and handed another. + + Two things are worth knowing before you use it. A UTF-16 file stores two + bytes for every character, so it always has an even number of them and an odd + size is refused - the refusal names the nearest size above and below that it + can write. And the smallest XML file grows from 264 B to 532 B, because the + declaration, the root element and one whole record all cost twice as much. + + `encoding=utf-16le` and `encoding=utf-16be` need `bom=true`. The XML + specification requires a mark on a UTF-16 document, and asking for one + without it is refused rather than written. + - **SVG drawings can be any size you ask for.** Two new settings on `svg`: `width` and `height`, both a whole number of pixels from 1 to 20000. They default to the 800 by 600 these drawings have always been, so a recipe that @@ -375,6 +394,29 @@ because it turns other people's test suites red. ### Fixed +- **A size range no longer fails on sizes the format cannot write.** Some + formats cannot produce every byte count. A file written in UTF-16 always has + an even number of bytes, so half of any range was unreachable, and a picture + cannot use the handful of byte counts just above its encoded size, because the + smallest padding it can add costs more than that. + + Until now a size drawn onto one of those was refused, and the whole run + stopped. Whether that happened depended on the seed and the number of files, + so the same recipe worked one day and not the next. + + A range asks for some size between two ends rather than for a number, so a + drawn size the format cannot write is now moved to the nearest one it can, + inside the range that was asked for. Files that moved carry a `size_moved` + note in the manifest, and the run says so once. + + Two things are deliberately unchanged. A range that starts below what the + format can produce at all is still refused, naming the smallest size it can + write, because that is a recipe worth correcting rather than a run worth + quietly filling with identical files. And `--size 1001` still refuses, because + that named a number. + + Runs that worked before are byte for byte what they were. + - **The window now warns when a run's record will be too big for this build to read back.** The command line has said this since the ceiling was measured. The window said nothing at all, so somebody who generated 25 000 files from it diff --git a/README.md b/README.md index e2090e6..7aa4b5a 100644 --- a/README.md +++ b/README.md @@ -476,10 +476,10 @@ recipe. `tfg formats ` prints the allowed range or list for each: | `pptx` | `slides` | | `csv` | `delimiter`, `line_ending`, `header`, `quote_style`, `columns` | | `log` | `entry_format`, `timestamps`, `rate`, `methods`, `status_mix`, `level_mix`, `ip_version`, `line_ending` | -| `txt`, `md` | `encoding`, `bom` | +| `txt`, `md`, `xml` | `encoding`, `bom` | | `json` | `formatting` | | `svg` | `width`, `height` | -| `xml`, `html` | none | +| `html` | none | ``` tfg generate --format jpg --size 500kb --set width=1920 --set height=1080 --set quality=85 diff --git a/internal/engine/drawsizes.go b/internal/engine/drawsizes.go new file mode 100644 index 0000000..b366f85 --- /dev/null +++ b/internal/engine/drawsizes.go @@ -0,0 +1,171 @@ +package engine + +import ( + "errors" + + "github.com/donislawdev/TestingFilesGenerator/internal/core" + "github.com/donislawdev/TestingFilesGenerator/internal/format" +) + +// Settling the size of every file of a range target. +// +// Taken out of engine.go on 2026-09-08, when snapping arrived and that file +// went past the size ceiling. The line is what a part does rather than how long +// it is: everything here answers "what size does each file of this range get", +// and nothing else in the engine asks that question. + +// firstWritable is the smallest size at or above from that this format will +// really write, without going past limit. +// +// It JUMPS rather than scans, and that is the whole reason it is cheap enough +// to run for every file. A format refusing a size it cannot write already names +// the next one it can, in BelowMinimumError.Minimum, so one refusal is normally +// one step. Measured 2026-09-08 on the two shapes this project has: an odd size +// under UTF-16 answers with size+1, and a PNG size inside the band above its +// encoded picture answers with the top of that band - 143 B answers 154 B. +// +// When nothing in the span is writable it hands back the format's OWN refusal +// for the first size it tried. That refusal already carries the four parts a +// refusal owes and names a size that would work, so there is no second wording +// here to drift away from it. +// +// The judge is the generator itself rather than a second copy of its rules. A +// copy would be a place for the two to disagree, and the disagreement would +// surface as a size that drawing accepted and writing refused. +// +// Descriptor.SmallestAccepted is this function's sibling and walks the same +// jump, from zero and with no ceiling. They are NOT one function, and the reason +// is the crash guard: everything the engine asks of a generator goes through +// planWithoutCrashing, so a panic costs one file rather than the process, while +// SmallestAccepted is called from guards and from the window, where that wrapper +// is not in hand. Merging them would mean handing a planner in, which buys less +// than it costs. +func firstWritable(desc format.Descriptor, r format.Request, from, limit int64) (int64, error) { + var first error + for at := from; at <= limit; { + r.Bytes = at + _, err := planWithoutCrashing(desc, r) + if err == nil { + return at, nil + } + if first == nil { + first = err + } + var below *format.BelowMinimumError + if !errors.As(err, &below) || below.Minimum <= at { + // Not a refusal about the size, or one that names no way forward. + // Either way there is nothing to jump to. + return 0, err + } + at = below.Minimum + } + return 0, first +} + +// drawSizes settles the size of every file of a range target. +// +// Every size is settled here, before a single byte is written, and that order +// is the point rather than an optimisation. A tool whose whole promise is that +// the same seed gives the same run cannot have an error that appears and +// disappears with the count. +// +// A DRAWN SIZE IS SNAPPED to one the format can write, and that is what this +// function gained on 2026-09-08. Until then it drew from the interval and hoped: +// a size the format could not write was refused later, by the per file plan, and +// whether that happened depended on what came out of the seed. Two shapes cause +// it and neither is rare. Four formats have unreachable BANDS - PNG cannot use +// the eleven byte counts above a picture's encoded size, because the smallest +// padding chunk costs twelve, and the OPC three declare the same shape. Three +// more have unreachable PARITY - a UTF-16 file is a whole number of sixteen bit +// units, so half of every range is unwritable, which took `--size-range +// 1000-1010` down about half the time (O190). +// +// Snapping is not rounding, and rule 1 is untouched. A range is a request for +// SOME size between two ends, not for a number - so answering with a writable +// size inside it is the answer, while `--size 1001` still refuses because that +// one named a number. +// +// PER FILE, not once for the target, because writability moves with the SEED as +// well as with the size: the same 64x64 PNG recipe has a floor of 144, 143 and +// 144 B at seeds 1, 2 and 3. Judging file 0 says nothing about file 2, and this +// comment claimed otherwise until 2026-09-08. +// +// What was written here before, and is now obsolete: closing this "needs the +// format to declare its unreachable bands, which is a change to +// format.Descriptor and the owner's call". It needed no such thing. The refusal +// ALREADY carries the next writable size, so probing and jumping does it with +// no new surface - measured before the change rather than argued. +// +// Bytes do not move for any run that worked before. A run that succeeds today +// has every drawn size writable, or it would have failed, and snapping a +// writable size returns it unchanged. +func drawSizes(t *Target, desc format.Descriptor, targetSeed uint64) error { + first := format.Request{ + Contains: t.Contains, + Seed: core.FileSeed(targetSeed, 0), + Label: t.Label, + Properties: t.Properties, + } + + // The low end is judged against what this format can do AT ALL, and that + // check is older than the snapping below it. The two answer different + // questions and both are wanted. + // + // A range starting under the format's floor is a recipe somebody should + // fix: asking PDF for 10 B to 8 kB says a spread was wanted and most of it + // does not exist, so the honest answer is the format's own refusal naming + // its floor, not forty files quietly piled on it. A range starting at or + // above the floor whose SOME sizes are unwritable - odd numbers under + // UTF-16, the band above a PNG's encoded picture - is a different thing: + // nobody can be expected to enumerate those, and snapping inside the range + // is the answer. + floor, err := firstWritable(desc, first, 0, t.SizeMax) + if err != nil { + // The floor is above the whole range, so nothing in it is writable. + return err + } + if t.SizeMin < floor { + // Asked again at the low end so the format words its own refusal, with + // the number the person actually wrote. + first.Bytes = t.SizeMin + if _, err := planWithoutCrashing(desc, first); err != nil { + return err + } + } + + span := uint64(t.SizeMax - t.SizeMin) + t.SizeMoved = make([]bool, len(t.Sizes)) + + for i := range t.Sizes { + want := t.SizeMin + if span != 0 { + // Per index, never from a running stream. Raising a count then + // leaves the sizes of the earlier files alone, which is rule 2 and + // the reason core.SizeSeed takes an index at all. + r := core.NewRand(core.SizeSeed(targetSeed, i)) + want = t.SizeMin + int64(r.Uint64N(span+1)) + } + + req := format.Request{ + Contains: t.Contains, + Seed: core.FileSeed(targetSeed, i), + Label: t.Label, + Properties: t.Properties, + } + + got, err := firstWritable(desc, req, want, t.SizeMax) + if err != nil && want > t.SizeMin { + // Nothing writable from the draw upwards. The bottom of the range + // can still hold something - a draw landing on the last odd number + // of a range has nowhere above it and plenty below - so the range is + // only empty once THAT fails too. + got, err = firstWritable(desc, req, t.SizeMin, t.SizeMax) + } + if err != nil { + return err + } + t.SizeMoved[i] = got != want + t.Sizes[i] = got + } + return nil +} diff --git a/internal/engine/engine.go b/internal/engine/engine.go index 73bb581..9ac0b24 100644 --- a/internal/engine/engine.go +++ b/internal/engine/engine.go @@ -51,6 +51,15 @@ type Target struct { SizeIsRange bool SizeMin int64 SizeMax int64 + // SizeMoved marks the files whose drawn size was not one the format can + // write, so the nearest writable one was used instead. Empty for a target + // that is not a range. + // + // Kept so the move can be REPORTED rather than done quietly. Silence is + // banned, and a person who asked for a spread and got one size back should + // not have to find that out by reading a manifest they had no reason to + // open. + SizeMoved []bool // BoundaryLimit is the limit a boundary set was built around, zero when // this target is not one. The three files name themselves from it. // @@ -70,71 +79,6 @@ type Target struct { Properties map[string]string } -// drawSizes settles the size of every file of a range target. -// -// Judged at the low end before a single size is drawn, and that order is the -// point rather than an optimisation. A range whose low end the format cannot -// deliver - below the minimum of the format, or too small to hold what the -// container was told to hold - would otherwise fail on some runs and not -// others, depending on what came out of the seed. A tool whose whole promise -// is that the same seed gives the same run cannot have an error that appears -// and disappears. -// -// THE LOW END IS NOT THE WHOLE ANSWER, and this comment claimed it was -// until 2026-09-06. It said the range "either works for every file or for -// none", and the code does not provide that. The check here is sufficient only -// if a format's reachable sizes are one unbroken interval starting at its -// minimum, and for four of them they are not: PNG has an unreachable band of -// eleven byte counts immediately above every picture's encoded size, because -// the smallest padding chunk costs twelve bytes, and the OPC three declare the -// same shape between the comment capacity and the smallest extra part. -// -// So a size DRAWN into such a band is refused later, by the per file plan, and -// whether that happens depends on the count. Measured on 2026-09-06, one 64x64 -// PNG recipe at one seed with size-range 143-200: counts 1 and 2 are accepted, -// counts 3, 5, 8, 12, 20 and 40 are refused. The low end moves with the seed -// too - 144, 143, 144 B at seeds 1, 2 and 3 - so judging file 0's band says -// nothing about file 2's. -// -// The bytes are stable under a raised count and that was verified, so rule 2 -// holds for CONTENT. What is not stable is whether the run happens at all. -// Closing that needs the format to declare its unreachable bands so the whole -// interval can be judged before anything is drawn, which is a change to -// format.Descriptor and the owner's call. Until then the refusal at least -// names the key the recipe carries - see atTarget - rather than pointing at a -// "size" setting a range target does not have. -// -// The judge is the generator itself rather than a second copy of its rules -// here. A copy would be a place for the two to disagree, and the disagreement -// would surface as a file that planning accepted and writing refused. -func drawSizes(t *Target, desc format.Descriptor, targetSeed uint64) error { - if _, err := planWithoutCrashing(desc, format.Request{ - Bytes: t.SizeMin, - Contains: t.Contains, - Seed: core.FileSeed(targetSeed, 0), - Label: t.Label, - Properties: t.Properties, - }); err != nil { - return err - } - - span := uint64(t.SizeMax - t.SizeMin) - for i := range t.Sizes { - if span == 0 { - // Both ends the same is legal and means identical files. Drawing - // from a range of one is not wrong, it just reads worse. - t.Sizes[i] = t.SizeMin - continue - } - // Per index, never from a running stream. Raising a count then leaves - // the sizes of the earlier files alone, which is rule 2 and the reason - // core.SizeSeed takes an index at all. - r := core.NewRand(core.SizeSeed(targetSeed, i)) - t.Sizes[i] = t.SizeMin + int64(r.Uint64N(span+1)) - } - return nil -} - // Uniform is n files of the same size, which is what most targets ask for. func Uniform(n int, bytes int64) []int64 { if n <= 0 { diff --git a/internal/engine/plantarget.go b/internal/engine/plantarget.go index f47162b..a9f0d9c 100644 --- a/internal/engine/plantarget.go +++ b/internal/engine/plantarget.go @@ -47,6 +47,21 @@ func (pl *planning) files(ctx context.Context, t *Target, desc format.Descriptor return atTarget(position, t, err) } + // A size drawn from the range that this format cannot write was moved + // to the nearest one it can. Silence is banned, so it is said here. + // + // The wording carries NO number on purpose. Notes are grouped by their + // text, so a number would make every file its own line - 25 000 of them + // on a big run, which is what buried the one note that mattered before + // grouping arrived. Which files moved is still exact in the manifest, + // because the note sits on each of their entries. + if idx < len(t.SizeMoved) && t.SizeMoved[idx] { + p.Notes = append(p.Notes, format.Note{ + Code: "size_moved", + Detail: "A size drawn from the range is not one this format can write, so the nearest size it can write was used instead. The file is still inside the range that was asked for.", + }) + } + name, err := renderName(t, desc, idx) if err != nil { return atTarget(position, t, err) diff --git a/internal/format/xmlfile/xml.go b/internal/format/xmlfile/xml.go index 33113f7..fa6f71f 100644 --- a/internal/format/xmlfile/xml.go +++ b/internal/format/xmlfile/xml.go @@ -18,6 +18,7 @@ import ( "github.com/donislawdev/TestingFilesGenerator/internal/core" "github.com/donislawdev/TestingFilesGenerator/internal/format" + "github.com/donislawdev/TestingFilesGenerator/internal/format/textenc" ) // Measured on 2026-08-01, a comment holds arbitrary bytes to 1 MiB both in the @@ -41,9 +42,13 @@ import ( const ( generatorVersion = "1" - declaration = `` + "\n" - rootOpen = "\n" - rootClose = "\n" + // The declaration names the encoding the bytes are really in, so the two + // move together. UTF-16LE and UTF-16BE share one spelling because the byte + // order mark is what tells them apart, and one is always written. + declarationUTF8 = `` + "\n" + declarationUTF16 = `` + "\n" + rootOpen = "\n" + rootClose = "\n" emailDomain = "@example.com" createdDate = "2026-08-01" @@ -102,8 +107,9 @@ func init() { Label: format.LabelInternal, Oracle: "python-xml", // Depth, element counts, namespaces, CDATA and an internal DTD come - // later. Declaring none now makes a recipe asking for them fail loudly. - Properties: nil, + // later. Declaring only what is here is what makes a recipe asking for + // them fail loudly instead of quietly producing something else. + Properties: textenc.Properties(), GeneratorVersion: generatorVersion, Generator: generator{}, }) @@ -114,10 +120,24 @@ type generator struct{} type memo struct { comment string // includes the trailing newline, empty when absent seed uint64 + codec textenc.Codec + // source is how many characters of ASCII the document holds, which is the + // ordered size less the mark and divided by the width of a character. + // Everything below counts in these rather than in file bytes, so the + // filling loop is the loop it always was. + source int64 } func (generator) Plan(r format.Request) (format.Plan, error) { - min := minimumBytes() + codec, err := textenc.Parse("xml", r.Properties) + if err != nil { + return format.Plan{}, err + } + if err := checkMark(codec); err != nil { + return format.Plan{}, err + } + + min := minimumFor(codec) if r.Bytes < min { return format.Plan{}, &format.BelowMinimumError{ Format: "XML", @@ -127,35 +147,43 @@ func (generator) Plan(r format.Request) (format.Plan, error) { Hint: fmt.Sprintf("Ask for %d B or more.", min), } } + // Half of all sizes are unreachable in UTF-16, and a refusal has to name + // the nearest reachable one in both directions rather than say no. + if err := codec.Check("XML", r.Bytes); err != nil { + return format.Plan{}, err + } p := format.Plan{ Bytes: r.Bytes, Exact: true, Determinism: format.DeterminismByte, Properties: map[string]any{ - "encoding": "utf-8", - "line_ending": "lf", - "root": "records", - "declaration": true, + textenc.Setting: codec.Name(), + textenc.SettingBOM: codec.HasBOM(), + "line_ending": "lf", + "root": "records", + "declaration": true, }, } - m := memo{seed: r.Seed} + m := memo{seed: r.Seed, codec: codec, source: codec.Source(r.Bytes)} if r.Label { // A comment carries the label without touching the content, and it can // sit anywhere after the declaration. The label text uses spaced // hyphens and never a double one, which a comment may not contain. line := "\n" // It has to leave room for a whole document beside it, or the file would - // be a comment and an empty root. - if int64(len(line))+minimumBytes() <= r.Bytes { + // be a comment and an empty root. The number is what the comment COSTS + // in this encoding, not how long it is to read - in UTF-16 those differ + // by a factor of two, and a note off by half is worse than no note. + if codec.Cost(int64(len(line)))+min <= r.Bytes { m.comment = line } else { p.Notes = append(p.Notes, format.Note{ Code: "label_omitted", Detail: fmt.Sprintf( "The label comment needs %d B and this file has no room for it beside a whole record. Its name and the manifest still identify it.", - len(line)), + codec.Cost(int64(len(line)))), }) } } @@ -171,13 +199,20 @@ func (generator) Write(ctx context.Context, w io.Writer, p format.Plan) error { return fmt.Errorf("xml: the plan was not produced by this generator") } - head := declaration + m.comment + rootOpen + // The mark is bytes rather than text, so it goes out as itself. Everything + // after it is characters, so it goes through the encoder. + if err := core.WriteAll(w, m.codec.Preamble()); err != nil { + return err + } + w = m.codec.Writer(w) + + head := declarationFor(m.codec) + m.comment + rootOpen if err := core.WriteAll(w, []byte(head)); err != nil { return err } rng := core.NewRand(m.seed) - return core.FillRecords(ctx, w, rng, p.Bytes-int64(len(head)), &records{}) + return core.FillRecords(ctx, w, rng, m.source-int64(len(head)), &records{}) } // records builds the record elements. It carries the record number, so the id @@ -297,12 +332,50 @@ func appendFiller(dst []byte, n int64) []byte { return core.AppendFiller(dst, words, n, nil) } -// minimumBytes is the declaration, the root element and one whole record, -// computed rather than written down so it cannot drift away from the template -// the way a number in a document would. -func minimumBytes() int64 { +// declarationFor is the opening line for one encoding. It has to name what the +// bytes really are: a declaration that disagrees with them is the one defect an +// outside reader catches on its own - expat refuses it in both directions, +// measured 2026-09-08 - so this is the single place the two are kept in step. +func declarationFor(c textenc.Codec) string { + if c.Name() == textenc.UTF8 { + return declarationUTF8 + } + return declarationUTF16 +} + +// checkMark refuses UTF-16 without a byte order mark. +// +// The specification requires one for a UTF-16 entity and this format is +// declared at full fidelity, so writing a document without it would be a breach +// nothing here would notice: expat ACCEPTS such a file and reads it correctly, +// measured 2026-09-08, so the oracle cannot go red on it. A file that breaks +// the specification on purpose belongs to the chaos lab, which does not exist. +func checkMark(c textenc.Codec) error { + if c.Name() == textenc.UTF8 || c.HasBOM() { + return nil + } + return &format.PropertyValueError{ + Format: "xml", + Key: textenc.SettingBOM, + Value: "false", + Reason: "XML in " + c.Name() + " has to open with a byte order mark, so this needs bom=true or encoding=" + textenc.UTF8, + } +} + +// minimumBytes is the smallest file in the default encoding, which is the one +// the registry declares. +func minimumBytes() int64 { return minimumFor(textenc.Default()) } + +// minimumFor is the declaration, the root element and one whole record in one +// encoding, computed rather than written down so it cannot drift away from the +// template the way a number in a document would. +// +// Every encoding answers for its own minimum and the registry declares only the +// default one, which is what JSON settled on when its layouts did the same. +func minimumFor(c textenc.Codec) int64 { var r records - return int64(len(declaration)+len(rootOpen)) + r.Shortest() + source := int64(len(declarationFor(c))+len(rootOpen)) + r.Shortest() + return c.Cost(source) + int64(len(c.Preamble())) } // longestWord and longestVendor are the widest draws, because the minimum has diff --git a/internal/guard/layers_test.go b/internal/guard/layers_test.go index 884f711..1740cf4 100644 --- a/internal/guard/layers_test.go +++ b/internal/guard/layers_test.go @@ -122,7 +122,7 @@ var sameLayerAllowed = map[string][]string{ "internal/format/logfile": {"internal/format"}, "internal/format/csvfile": {"internal/format"}, "internal/format/jsonfile": {"internal/format"}, - "internal/format/xmlfile": {"internal/format"}, + "internal/format/xmlfile": {"internal/format", "internal/format/textenc"}, "internal/format/htmlfile": {"internal/format"}, "internal/format/svgfile": {"internal/format"}, "internal/format/bmp": {"internal/format", "internal/format/imagelabel"}, diff --git a/internal/guard/parity_test.go b/internal/guard/parity_test.go index ab34989..7bb1694 100644 --- a/internal/guard/parity_test.go +++ b/internal/guard/parity_test.go @@ -129,6 +129,8 @@ var reachableFromTheWindow = []string{ "property:log.timestamps", "property:md.bom", "property:md.encoding", + "property:xml.bom", + "property:xml.encoding", "property:pdf.page_size", "property:pdf.pages", "property:png.height", diff --git a/internal/guard/rangesnap_test.go b/internal/guard/rangesnap_test.go new file mode 100644 index 0000000..0f3dfd0 --- /dev/null +++ b/internal/guard/rangesnap_test.go @@ -0,0 +1,353 @@ +package guard + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/donislawdev/TestingFilesGenerator/internal/cli" + "github.com/donislawdev/TestingFilesGenerator/internal/core" + "github.com/donislawdev/TestingFilesGenerator/internal/format" +) + +// A range asks for SOME size between two ends, not for a number, so a size the +// format cannot write is moved to the nearest one it can rather than refused. +// +// This is O190, and it was worth fixing because it took whole runs down for a +// reason nobody could see coming. Two shapes cause it. A UTF-16 file is a whole +// number of sixteen bit units, so half of every range is unwritable and +// `--size-range 1000-1010` failed about half the time. Four formats have +// unreachable BANDS instead - PNG cannot use the eleven byte counts above a +// picture's encoded size, because the smallest padding chunk costs twelve. +// +// The line between snapping and refusing is the point of these guards, and +// getting it wrong in either direction is a real defect: +// +// - the low end UNDER the format's floor stays a refusal. Asking PDF for 10 B +// to 8 kB says a spread was wanted and most of it does not exist, so the +// answer is the format's own refusal, not forty files piled on the floor. +// TestARangeIsJudgedByItsLowEndRatherThanByWhatWasDrawn holds that half. +// - a size unwritable INSIDE what the format can do gets snapped. Nobody can +// be expected to enumerate parities and bands. +// +// See docs/OBSERVATIONS.md O190. + +// pictureFloor is the smallest PNG this recipe can be asked for, worked out +// rather than written down. +// +// It has to be worked out, and the first version of this file learned that the +// expensive way: the number moves with the SEED, and the seed a file gets is +// derived through the target's id, so the floor under a recipe is not the floor +// under the same seed on the command line. A hardcoded 143 passed by hand and +// failed here, which is the guard reaching a state it assumed rather than +// asserted - O118, from the inside. +// +// The band sits immediately above it, so a range has to START here to hold one. +func pictureFloor(t *testing.T, seed int64, id string) int64 { + t.Helper() + d, err := format.Get("png") + if err != nil { + t.Fatal(err) + } + return d.SmallestAccepted(format.Request{ + Seed: core.FileSeed(core.TargetSeed(seed, id), 0), + Label: true, + Properties: map[string]string{"width": "64", "height": "64"}, + }) +} + +// snapCases are the two shapes, named rather than derived so a third arriving +// without a case here is a gap somebody has to notice. +func snapCases(t *testing.T) []struct { + name string + body string + step int64 // sizes have to be a multiple of this, 1 when anything goes + count int +} { + t.Helper() + floor := pictureFloor(t, 2, "a") + + return []struct { + name string + body string + step int64 + count int + }{ + { + // Parity. Every size in this range is legal for the format and half + // of them are unwritable in this encoding. + name: "a wide encoding makes half the range unwritable", + body: `version: 1 +seed: 3 +targets: + - id: a + format: xml + count: 6 + size-range: 1001-1200 + properties: + encoding: utf-16le + bom: true +`, + step: 2, + count: 6, + }, + { + // A band. The floor itself is writable and the eleven byte counts + // above it are not, because the padding chunk that makes up any + // difference costs twelve. The range starts at the floor so the band + // is inside it, and reaches well past the band so there is something + // to snap TO. + name: "a band above the encoded picture is unwritable", + body: fmt.Sprintf(`version: 1 +seed: 2 +targets: + - id: a + format: png + count: 12 + size-range: %d-%d + properties: + width: 64 + height: 64 +`, floor, floor+57), + step: 1, + count: 12, + }, + } +} + +// TestASizeTheFormatCannotWriteIsMovedInsideTheRange is the claim itself. +func TestASizeTheFormatCannotWriteIsMovedInsideTheRange(t *testing.T) { + for _, c := range snapCases(t) { + t.Run(c.name, func(t *testing.T) { + files, notes := generateAndRead(t, c.body) + if len(files) != c.count { + t.Fatalf("produced %d files, expected %d", len(files), c.count) + } + + // Asserted rather than assumed, and this is the half that makes the + // rest mean anything. A range whose draws all happened to miss the + // unwritable sizes would pass every check below without one byte + // ever being snapped - green, and about nothing. + if !notes["size_moved"] { + t.Fatalf("no size had to move in this run, so it is not exercising the thing it names") + } + + lo, hi := rangeEnds(t, c.body) + for _, f := range files { + if f.bytes < lo || f.bytes > hi { + t.Errorf("%s came out at %d B, outside the %d to %d that was asked for", + f.name, f.bytes, lo, hi) + } + if c.step > 1 && f.bytes%c.step != 0 { + t.Errorf("%s came out at %d B, which this encoding cannot write", + f.name, f.bytes) + } + } + + // The control, and without it a build that answered every range + // with one size would pass everything above. The whole point of a + // range is that the sizes differ. + seen := map[int64]bool{} + for _, f := range files { + seen[f.bytes] = true + } + if len(seen) < 2 { + t.Errorf("all %d files came out the same size, so snapping has flattened the range", + len(files)) + } + }) + } +} + +// TestAMovedSizeIsReported holds rule 6 against the fix itself. +// +// Snapping is the right answer and it is still a thing the tool did that nobody +// asked for by name. Silence is banned, so it is said - and the control beside +// it is the half that makes this mean something: a range where nothing moves +// has to stay quiet, or the note would be noise on every run. +func TestAMovedSizeIsReported(t *testing.T) { + moved := notesFromRun(t, `version: 1 +seed: 3 +targets: + - id: a + format: xml + count: 4 + size-range: 1001-1200 + properties: + encoding: utf-16le + bom: true +`) + if !moved["size_moved"] { + t.Error("sizes were moved to fit the format and nothing said so - silence is banned") + } + + quiet := notesFromRun(t, `version: 1 +seed: 3 +targets: + - id: a + format: xml + count: 4 + size-range: 1000-1200 + properties: + encoding: utf-8 + bom: false +`) + if quiet["size_moved"] { + t.Error("nothing had to move in this run and it was reported anyway, which would make the note noise") + } +} + +// TestARangeHoldingNoWritableSizeIsRefusedBeforeAnyFile is the other end. +// +// Snapping cannot invent a size that is not there. Both shapes get a case, +// because a range sitting entirely inside a band and a range holding only odd +// numbers fail for different reasons in the format and have to come out the +// same way here. +func TestARangeHoldingNoWritableSizeIsRefusedBeforeAnyFile(t *testing.T) { + cases := []struct{ name, body string }{ + {"only odd sizes in a wide encoding", `version: 1 +seed: 3 +targets: + - id: a + format: xml + count: 2 + size-range: 1001-1001 + properties: + encoding: utf-16le + bom: true +`}, + {"entirely inside a band", `version: 1 +seed: 2 +targets: + - id: a + format: png + count: 4 + size-range: 143-153 + properties: + width: 64 + height: 64 +`}, + } + + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + dir := t.TempDir() + out := filepath.Join(dir, "out") + path := writeRecipe(t, dir, c.body) + + code, stdout, errOut := run(t, "generate", path, "--out", out) + if code != cli.ExitFormat { + t.Fatalf("exit %d, expected %d - no size in this range can be written:\n%s", + code, cli.ExitFormat, errOut) + } + if stdout != "" { + t.Errorf("a failed run wrote to stdout:\n%s", stdout) + } + if n := len(filesIn(t, out)); n != 0 { + t.Errorf("%d file(s) were written by a run that was refused", n) + } + }) + } +} + +// TestSnappingStillLeavesTheEarlierFilesAlone is rule 2 applied to the fix. +// +// Snapping is a pure function of the drawn size, and the drawn size comes from +// the index - so raising the count cannot move a file that was already there. +// A build that snapped by walking forward from wherever the last file landed +// would pass every other guard here and break this one. +func TestSnappingStillLeavesTheEarlierFilesAlone(t *testing.T) { + const body = `version: 1 +seed: 3 +targets: + - id: a + format: xml + count: %s + size-range: 1001-1200 + properties: + encoding: utf-16le + bom: true +` + three := generateInto(t, strings.Replace(body, "%s", "3", 1)) + nine := generateInto(t, strings.Replace(body, "%s", "9", 1)) + + if len(three) != 3 || len(nine) != 9 { + t.Fatalf("produced %d and %d files, expected 3 and 9", len(three), len(nine)) + } + for i := range three { + if three[i].bytes != nine[i].bytes || three[i].sha != nine[i].sha { + t.Errorf("file %d differs between a count of 3 and a count of 9: %d B %s against %d B %s", + i+1, three[i].bytes, three[i].sha[:12], nine[i].bytes, nine[i].sha[:12]) + } + } +} + +// rangeEnds reads the two ends back out of the recipe, so the numbers this +// checks against are the ones that were asked for rather than a second copy. +func rangeEnds(t *testing.T, body string) (int64, int64) { + t.Helper() + for _, line := range strings.Split(body, "\n") { + line = strings.TrimSpace(line) + if !strings.HasPrefix(line, "size-range:") { + continue + } + lo, hi, err := core.ParseSizeRange(strings.TrimSpace(strings.TrimPrefix(line, "size-range:"))) + if err != nil { + t.Fatalf("reading the range out of the recipe: %v", err) + } + return lo, hi + } + t.Fatal("this recipe carries no size-range, so this check is reading the wrong thing") + return 0, 0 +} + +// notesFromRun generates and hands back the note codes the manifest carries. +func notesFromRun(t *testing.T, body string) map[string]bool { + t.Helper() + _, notes := generateAndRead(t, body) + return notes +} + +// generateAndRead runs one recipe and reads back both what landed on the disk +// and what the manifest says about it. +// +// The files come off the DISK rather than out of the manifest, because the +// manifest is this tool describing its own work and half of these checks ask +// what a person actually got. The notes have to come from the manifest, since +// that is the only place they are written down per file. +func generateAndRead(t *testing.T, body string) ([]fileFact, map[string]bool) { + t.Helper() + dir := t.TempDir() + out := filepath.Join(dir, "out") + path := writeRecipe(t, dir, body) + + code, _, errOut := run(t, "generate", path, "--out", out) + if code != cli.ExitOK { + t.Fatalf("exit %d:\n%s", code, errOut) + } + + raw, err := os.ReadFile(filepath.Join(out, "manifest.json")) + if err != nil { + t.Fatalf("reading the manifest: %v", err) + } + var doc struct { + Files []struct { + Notes []struct { + Code string `json:"code"` + } `json:"notes"` + } `json:"files"` + } + if err := json.Unmarshal(raw, &doc); err != nil { + t.Fatalf("parsing the manifest: %v", err) + } + codes := map[string]bool{} + for _, f := range doc.Files { + for _, n := range f.Notes { + codes[n.Code] = true + } + } + return describeFiles(t, out), codes +} diff --git a/internal/guard/textencoding_test.go b/internal/guard/textencoding_test.go index 2aa3ef4..ac3df9b 100644 --- a/internal/guard/textencoding_test.go +++ b/internal/guard/textencoding_test.go @@ -34,7 +34,42 @@ import ( // encodedFormats is the formats that take an encoding, named rather than // derived - so a third one arriving without being added here is a gap somebody // has to notice rather than a loop that quietly gets shorter. -var encodedFormats = []string{"txt", "md"} +var encodedFormats = []string{"txt", "md", "xml"} + +// markRequired is the formats where a wide encoding has to carry a byte order +// mark, so the two cases without one are refused rather than written. +// +// XML is here for a measured reason rather than a tidy one. Its specification +// requires a mark on a UTF-16 entity, and - the half that decided it - our +// oracle cannot go red on a document that lacks one: expat ACCEPTS such a file +// and reads it correctly, measured 2026-09-08 in both directions. A file +// nothing here could check is a file this tool does not write. The refusal +// itself is proven by TestXMLRefusesAWideEncodingWithoutAMark, so this map is +// a declared behaviour rather than a way of skipping cases. +var markRequired = map[string]bool{"xml": true} + +// refusedBy says whether this format turns this combination down by design. +func (c encodingCase) refusedBy(id string) bool { + return markRequired[id] && c.width == 2 && !c.bom +} + +// labelLine is how one format writes a label, which the note's arithmetic +// depends on and no format exposes. A format missing from here stops the check +// rather than silently measuring an empty wrapper. +func labelLine(t *testing.T, id string, size int64, seed uint64) string { + t.Helper() + body := core.Label(id, size, seed) + switch id { + case "txt": + return body + "\n" + case "md": + return body + "\n\n" + case "xml": + return "\n" + } + t.Fatalf("%s takes an encoding and this check does not know how it wraps a label", id) + return "" +} type encodingCase struct { encoding string @@ -105,7 +140,7 @@ func writeEncoded(t *testing.T, id string, size int64, props map[string]string) // one question this has to answer. func TestATextFileIsTheEncodingItDeclares(t *testing.T) { dir := t.TempDir() - checked, skipped := 0, 0 + checked, skipped, refused := 0, 0, 0 for _, id := range encodedFormats { d, err := format.Get(id) @@ -113,6 +148,10 @@ func TestATextFileIsTheEncodingItDeclares(t *testing.T) { t.Fatal(err) } for _, c := range encodingCases() { + if c.refusedBy(id) { + refused++ + continue + } smallest := d.SmallestAccepted(format.Request{Seed: 7741, Label: true, Properties: c.props()}) if c.width == 2 && smallest%2 != 0 { t.Errorf("%s %v: the smallest size it accepts is %d, which a two byte encoding cannot write", @@ -152,7 +191,10 @@ func TestATextFileIsTheEncodingItDeclares(t *testing.T) { if checked == 0 { t.Errorf("nothing was decoded by anything outside this package - %d case(s) skipped", skipped) } - t.Logf("%d file(s) decoded strictly by Python, %d skipped", checked, skipped) + // The count is here so a format quietly declaring every combination + // refused would show up as nothing being checked rather than as a pass. + t.Logf("%d file(s) decoded strictly by Python, %d skipped, %d combination(s) refused by design", + checked, skipped, refused) } // TestAWideEncodingRefusesAnOddSizeAndNamesOneItCanWrite is the refusal, and @@ -162,15 +204,22 @@ func TestATextFileIsTheEncodingItDeclares(t *testing.T) { // ENCODING rather than about the number. Without that half, a generator that // refused every odd size in every encoding would pass this. func TestAWideEncodingRefusesAnOddSizeAndNamesOneItCanWrite(t *testing.T) { - odd := []int64{4001, 65, 1235} - for _, id := range encodedFormats { d, err := format.Get(id) if err != nil { t.Fatal(err) } for _, c := range encodingCases() { - for _, size := range odd { + if c.refusedBy(id) { + continue + } + // The odd sizes are taken from the floor rather than written down. + // A format with a minimum of its own - XML holds a declaration, a + // root and one whole record - would refuse a small fixed number for + // being too SMALL, and the refusal under test would never be the + // one that fired. A wide floor is even, so each of these is odd. + base := d.SmallestAccepted(format.Request{Seed: 7741, Label: true, Properties: c.props()}) + for _, size := range []int64{base + 1, base + 235, base + 4001} { _, err := d.Generator.Plan(format.Request{ Bytes: size, Seed: 7741, Label: true, Properties: c.props()}) @@ -220,7 +269,12 @@ func TestAWideEncodingRefusesAnOddSizeAndNamesOneItCanWrite(t *testing.T) { // before the setting existed. func TestTheDefaultEncodingIsTheBytesTheseFormatsAlwaysWrote(t *testing.T) { for _, id := range encodedFormats { - for _, size := range []int64{0, 33, 4096} { + d, err := format.Get(id) + if err != nil { + t.Fatal(err) + } + floor := d.SmallestAccepted(format.Request{Seed: 7741, Label: true}) + for _, size := range []int64{floor, floor + 1, floor + 4096} { silent := writeEncoded(t, id, size, nil) spoken := writeEncoded(t, id, size, map[string]string{ textenc.Setting: textenc.UTF8, textenc.SettingBOM: "false"}) @@ -241,26 +295,61 @@ func TestTheDefaultEncodingIsTheBytesTheseFormatsAlwaysWrote(t *testing.T) { // out by a factor of two is worse than no note: it tells somebody to ask for // 66 B when the file needs 132. func TestALabelThatWillNotFitSaysWhatItWouldCost(t *testing.T) { - const size = int64(64) // below the label's cost in a wide encoding, even - tails := map[string]string{"txt": "\n", "md": "\n\n"} - for _, id := range encodedFormats { d, err := format.Get(id) if err != nil { t.Fatal(err) } - props := map[string]string{textenc.Setting: textenc.UTF16LE} + props := map[string]string{textenc.Setting: textenc.UTF16LE, textenc.SettingBOM: "true"} + floor := d.SmallestAccepted(format.Request{Seed: 7741, Label: true, Properties: props}) + + // The size has to sit in a window or half of this check discriminates + // nothing: at least as long as the label READS, and below twice that. + // Inside it the label fits the file and does not fit what the file + // HOLDS, which is the whole difference between the two comparisons a + // generator could make - and a mutation swapping them is in the set. + // + // Taking the floor alone was wrong and the mutation run said so rather + // than the reading: TXT and MD sit on a floor of almost nothing, so the + // floor lands BELOW the window and both comparisons agree there. + // + // XML is the other way round. Its smallest document is a declaration, a + // root and a whole record - about six times its label - so the window is + // under its floor and unreachable. There the floor is the size and only + // the cost half of this check applies, which is honest: XML asks a + // different question of its own fit, and its own mutation covers it. + reads := int64(len(labelLine(t, id, floor, 7741))) + size := floor + if size < reads { + size = reads + 2 + } + if size%2 != 0 { + size++ + } p, err := d.Generator.Plan(format.Request{ Bytes: size, Seed: 7741, Label: true, Properties: props}) if err != nil { t.Fatalf("%s: planning %d B in utf-16le: %v", id, size, err) } - line := core.Label(id, size, 7741) + tails[id] + line := labelLine(t, id, size, 7741) wide, narrow := int64(len(line))*2, int64(len(line)) - if wide <= size { - t.Fatalf("%s: the label costs %d B at %d B, so this case no longer sits below the threshold", - id, wide, size) + + // The control, and it replaces a precondition that stopped meaning + // anything. Comparing the label with the file size only works while a + // format has no floor of its own - XML's floor is six times its label, + // so that comparison would have called this case broken. What has to be + // true is that a threshold EXISTS: given room, the note goes away. + roomy := size + wide*2 + if roomy%2 != 0 { + roomy++ + } + if q, err := d.Generator.Plan(format.Request{ + Bytes: roomy, Seed: 7741, Label: true, Properties: props}); err != nil { + t.Fatalf("%s: planning %d B in utf-16le: %v", id, roomy, err) + } else if noteWithCode(q.Notes, "label_omitted") != nil { + t.Fatalf("%s: %d B has room for a %d B label and the note fired anyway, so this is not measuring a threshold", + id, roomy, wide) } note := noteWithCode(p.Notes, "label_omitted") diff --git a/internal/guard/xmlencoding_test.go b/internal/guard/xmlencoding_test.go new file mode 100644 index 0000000..15d2de2 --- /dev/null +++ b/internal/guard/xmlencoding_test.go @@ -0,0 +1,189 @@ +package guard + +import ( + "bytes" + "os" + "path/filepath" + "testing" + + "github.com/donislawdev/TestingFilesGenerator/internal/format" + "github.com/donislawdev/TestingFilesGenerator/internal/format/textenc" + "github.com/donislawdev/TestingFilesGenerator/internal/oracle" +) + +// XML gained an encoding on 2026-09-08, and it is the first format here whose +// file SAYS which encoding it is in. TXT and MD carry that fact only in their +// bytes, so nothing inside them can disagree with anything. An XML declaration +// can, and a document announcing UTF-8 while holding UTF-16 is the classic +// parser trap. +// +// The measurement that made this worth building: expat refuses the mismatch in +// both directions - "encoding specified in XML declaration is incorrect" - so +// there is a witness outside this project. These two guards are the half that +// runs without one. See docs/XML-ENCODING-2026-09-08.md. + +// asDeclared widens an ASCII string the way the encoding under test would, so +// the expected declaration is built rather than written out three times. +func asDeclared(s string, c encodingCase) []byte { + if c.width == 1 { + return []byte(s) + } + out := make([]byte, 0, len(s)*2) + for _, b := range []byte(s) { + if c.encoding == textenc.UTF16BE { + out = append(out, 0x00, b) + } else { + out = append(out, b, 0x00) + } + } + return out +} + +// TestTheXMLDeclarationNamesTheEncodingTheBytesAreIn is the whole point of the +// setting in one sentence. +// +// Deliberately not asking Python. The structural check does ask, and it is the +// stronger reader - but it is skipped wherever Python is missing, and the one +// thing this change can break should not be provable only on a machine that +// happens to have an interpreter. +// +// Built from the bytes rather than decoded, because a decoder would have to be +// told which encoding to expect and would then be agreeing with itself. +func TestTheXMLDeclarationNamesTheEncodingTheBytesAreIn(t *testing.T) { + d, err := format.Get("xml") + if err != nil { + t.Fatal(err) + } + + checked := 0 + for _, c := range encodingCases() { + if c.refusedBy("xml") { + continue + } + + want := `` + other := `` + if c.width == 2 { + want, other = other, want + } + + floor := d.SmallestAccepted(format.Request{Seed: 7741, Label: true, Properties: c.props()}) + for _, size := range []int64{floor, floor + int64(c.width)*512} { + body := writeEncoded(t, "xml", size, c.props()) + + head := append(append([]byte{}, c.mark...), asDeclared(want, c)...) + if !bytes.HasPrefix(body, head) { + t.Errorf("xml %v at %d B: the file does not open with %q in its own encoding, it opens with % x", + c, size, want, first(body, len(head))) + } + // The control. Without it a generator writing BOTH declarations, or + // one writing the right bytes for the wrong reason, would pass the + // line above. + wrong := append(append([]byte{}, c.mark...), asDeclared(other, c)...) + if bytes.HasPrefix(body, wrong) { + t.Errorf("xml %v at %d B: the declaration says %q and the bytes are %s", + c, size, other, c.encoding) + } + checked++ + } + } + + if checked == 0 { + t.Fatal("no combination was checked, so this guard proves nothing") + } + t.Logf("%d file(s) opened with a declaration matching their bytes", checked) +} + +// TestXMLRefusesAWideEncodingWithoutAMark proves the entry in markRequired is a +// behaviour rather than an excuse for skipping two cases. +// +// The specification requires a mark on a UTF-16 entity. The reason it is +// REFUSED rather than merely discouraged is that nothing here could catch it: +// expat accepts a UTF-16 document with no mark and reads it correctly, measured +// 2026-09-08, so the oracle beside this cannot go red on one. A file this tool +// cannot check is a file it does not write, and a fixture that breaks a +// specification on purpose belongs to the chaos lab, which does not exist yet. +func TestXMLRefusesAWideEncodingWithoutAMark(t *testing.T) { + d, err := format.Get("xml") + if err != nil { + t.Fatal(err) + } + + for _, name := range []string{textenc.UTF16LE, textenc.UTF16BE} { + props := map[string]string{textenc.Setting: name, textenc.SettingBOM: "false"} + _, err := d.Generator.Plan(format.Request{Bytes: 4096, Seed: 7741, Label: true, Properties: props}) + if err == nil { + t.Errorf("%s without a mark was accepted, and the specification requires one", name) + continue + } + + // The refusal has to name both halves. One saying only "bom cannot be + // false" sends somebody to turn a setting on without saying why, and + // one naming only the encoding hides which knob to reach for. + msg := err.Error() + for _, part := range []string{textenc.SettingBOM, name, "byte order mark"} { + if !bytes.Contains([]byte(msg), []byte(part)) { + t.Errorf("%s: the refusal does not mention %q: %s", name, part, msg) + } + } + + // The control: the same encoding WITH a mark is accepted, so the + // refusal is about the missing mark rather than about UTF-16. + props[textenc.SettingBOM] = "true" + if _, err := d.Generator.Plan(format.Request{ + Bytes: 4096, Seed: 7741, Label: true, Properties: props}); err != nil { + t.Errorf("%s with a mark was refused too, so the refusal is not about the mark: %v", name, err) + } + } +} + +// TestTheStructuralCheckRefusesADeclarationThatDisagreesWithTheBytes is a +// canary, and it exists because of what the check beside it cannot prove. +// +// Every other guard here hands the structural check a CORRECT file, so all of +// them would stay green if that check stopped looking at the declaration +// entirely. That is the worst shape a mutation takes in this project: the +// pattern is found, the code compiles, and the broken text never reaches an +// assertion. The only way to know the check works is to hand it something +// wrong and watch it refuse. +// +// A rewritten declaration rather than a differently generated file, because +// this tool cannot be asked to produce one - which is the point. +func TestTheStructuralCheckRefusesADeclarationThatDisagreesWithTheBytes(t *testing.T) { + dir := t.TempDir() + told := []string{textenc.Setting + "=" + textenc.UTF8, textenc.SettingBOM + "=false"} + + body := writeEncoded(t, "xml", 4096, map[string]string{ + textenc.Setting: textenc.UTF8, textenc.SettingBOM: "false"}) + + truth := []byte(``) + lie := []byte(``) + if !bytes.HasPrefix(body, truth) { + t.Fatalf("this canary rewrites the declaration and the file does not open with the one it expects: % x", + first(body, len(truth))) + } + lying := append(append([]byte{}, lie...), body[len(truth):]...) + + honest := filepath.Join(dir, "honest.xml") + broken := filepath.Join(dir, "lying.xml") + if err := os.WriteFile(honest, body, 0o600); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(broken, lying, 0o600); err != nil { + t.Fatal(err) + } + + // The control first. A check that refused everything would pass the half + // below without seeing anything at all. + res := oracle.Strict("xml", honest, told...) + if !res.Available { + t.Skip("the structural check needs python") + } + if res.Err != nil { + t.Fatalf("the untouched file was refused, so this canary is measuring something else: %v", res.Err) + } + + if res := oracle.Strict("xml", broken, told...); res.Err == nil { + t.Error("a document holding UTF-8 while announcing UTF-16 was called correct, so nothing here can see the one defect this setting is able to cause") + } +} diff --git a/internal/oracle/strict.py b/internal/oracle/strict.py index d83aff6..033b8c8 100644 --- a/internal/oracle/strict.py +++ b/internal/oracle/strict.py @@ -511,7 +511,7 @@ def check_json(data, settings=None): ok(f"{len(doc)} records, keys {sorted(keys)}") -def scan_xml(data): +def scan_xml(text): """Well formed, checked by hand against the XML specification. Deliberately not expat. That is the reference tool beside this one, so @@ -522,12 +522,11 @@ def scan_xml(data): never contains a double hyphen, and text where every ampersand starts a real entity reference. A raw ampersand is the classic way a generated document stops being well formed while staying exactly the right size. - """ - try: - text = data.decode("utf-8") - except UnicodeDecodeError as exc: - fail(f"not valid UTF-8: {exc}") + Takes text rather than bytes since XML gained an encoding on 2026-09-08. A + scanner that decoded as UTF-8 itself would refuse every UTF-16 document, and + the caller is the one that knows which encoding was ordered. + """ if not text.startswith("', text) + if not opening: + fail("the document does not open with an XML declaration naming an encoding") + if opening.group(1) != want: + fail(f"the bytes are {ordered} and the declaration says {opening.group(1)!r}, " + f"so a reader is told one thing and handed another") + + elements, _ = scan_xml(text) if elements < 2: fail(f"the document holds {elements} element(s), so there is nothing below the root") - ok(f"{elements} elements, all balanced") + ok(f"{elements} elements, all balanced, declaration agrees with the bytes") def check_svg(data): @@ -615,9 +635,12 @@ def check_svg(data): shape, and a generator that quietly stopped emitting them would be the right size and would still open. """ - elements, seen = scan_xml(data) + try: + text = data.decode("utf-8") + except UnicodeDecodeError as exc: + fail(f"not valid UTF-8: {exc}") + elements, seen = scan_xml(text) - text = data.decode("utf-8") if "Settings each format accepts columns 1 - 64 columns + + xml + encoding + utf-16be, utf-16le, utf-8 + + + + bom + true or false + zip entries diff --git a/web/public/pl/formaty/index.html b/web/public/pl/formaty/index.html index 3d0a921..c5c5d94 100644 --- a/web/public/pl/formaty/index.html +++ b/web/public/pl/formaty/index.html @@ -620,6 +620,16 @@

Ustawienia, które przyjmuje każdy format

columns 1 - 64 kolumn + + xml + encoding + utf-16be, utf-16le, utf-8 + + + + bom + prawda albo fałsz + zip entries