From 23b0aeadd0dc1853e9dcdcedc17788444f0b6d26 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Tue, 15 Sep 2026 11:05:55 +0200 Subject: [PATCH 01/15] add print-on-demand edition of the PDF `task pdf:print` builds the book block for a printed book (BoD, A4, black and white) next to the screen PDF, as planned in tasks/spec.md. - mirrored page furniture, right-hand part starts, title page and imprint with an author list from `task pdf:authors` - links printed as page references and footnotes, images at 300 ppi without transparency, a black-and-white palette - tools/pdf/print.yml shortens the operator reference and the release notes to overview tables; reference categories print as one table - long code lines wrap inside their frame - both editions: tall figures scale to the page, image figures carry no Typst number - fix the transformer reference table, which ended at the Excel map row: the generator turns a fenced block in a description into a code span --- Taskfile.yml | 37 ++ .../reference/transformer/Replace/excelMap.md | 2 +- docs/build/reference/transformer/index.md | 3 +- poetry.lock | 127 +++- pyproject.toml | 1 + tasks/backlog.md | 428 ++++++++------ tasks/spec.md | 558 +++++++++++------- tests/test_build_pdf.py | 13 +- tests/test_build_pdf_images.py | 83 +++ tests/test_build_pdf_print.py | 248 ++++++++ tests/test_pdf_authors.py | 45 ++ tests/test_update_di_reference.py | 29 +- tools/__init__.py | 2 + tools/build_pdf.py | 555 +++++++++++++++-- tools/pdf/authors.yml | 49 ++ tools/pdf/filter.lua | 3 +- tools/pdf/print.yml | 30 + tools/pdf/style.typ | 312 ++++++++-- tools/pdf_authors.py | 104 ++++ tools/update_di_reference.py | 14 +- 20 files changed, 2152 insertions(+), 491 deletions(-) create mode 100644 tests/test_build_pdf_images.py create mode 100644 tests/test_build_pdf_print.py create mode 100644 tests/test_pdf_authors.py create mode 100644 tools/pdf/authors.yml create mode 100644 tools/pdf/print.yml create mode 100644 tools/pdf_authors.py diff --git a/Taskfile.yml b/Taskfile.yml index 837b9c496..93956aecf 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -200,6 +200,43 @@ tasks: cmds: - poetry run dec-tool build-pdf + pdf:print: + desc: Build the print edition of the PDF - the book block for print on demand + summary: | + Runs `task build`, then `dec-tool build-pdf --edition print`: the same + merge, pandoc run and fonts as `task pdf`, typeset as the book block of + a printed book (BoD, A4, black and white). tasks/spec.md describes it: + mirrored page furniture, right-hand part starts, title page and + imprint, links printed as page references and footnotes, and images + normalized to 300 ppi without transparency. + + tools/pdf/print.yml configures the edition: the publisher, and how each + navigation section is printed (full, list or omit). + tools/pdf/authors.yml is the imprint's author list; refresh it with + `task pdf:authors`. + + Writes dist/documentation-eccenca-com--print.pdf; the merged HTML, + the Typst source and the normalized images stay in dist/pdf/print/. The + screen PDF of `task pdf` is not affected. + deps: + - task: build + env: + BUILD_VERSION: '{{.CURRENT_VERSION}}' + cmds: + - poetry run dec-tool build-pdf --edition print + + pdf:authors: + desc: Refresh the author list of the print edition's imprint + summary: | + Writes tools/pdf/authors.yml from the GitHub contributors of this + repository: most commits first, without anonymous contributions, bot + accounts and software agents. Commit the result. Uses GITHUB_TOKEN or + GH_TOKEN when set; unauthenticated requests are rate limited. + deps: + - task: install + cmds: + - poetry run dec-tool pdf-authors + update:icons: desc: update the used eccenca icons from carbon cmds: diff --git a/docs/build/reference/transformer/Replace/excelMap.md b/docs/build/reference/transformer/Replace/excelMap.md index ae0489f48..a04d8873c 100644 --- a/docs/build/reference/transformer/Replace/excelMap.md +++ b/docs/build/reference/transformer/Replace/excelMap.md @@ -1,6 +1,6 @@ --- title: "Excel map" -description: "Replaces values based on a map of values read from a file in Open XML format (XLSX). The XLSX file may contain several sheets of the form: ```mapFrom,mapTo ,``` and more. An empty string can be created in Excel and alternatives by inserting `=''` in the input line of a cell. If there are multiple values for a single key, all values will be returned for the given key. Note that the mapping table will be cached in memory. If the Excel file is updated (even while transforming), the map will be reloaded within seconds." +description: "Replaces values based on a map of values read from a file in Open XML format (XLSX). The XLSX file may contain several sheets of the form: `mapFrom,mapTo ,` and more. An empty string can be created in Excel and alternatives by inserting `=''` in the input line of a cell. If there are multiple values for a single key, all values will be returned for the given key. Note that the mapping table will be cached in memory. If the Excel file is updated (even while transforming), the map will be reloaded within seconds." icon: octicons/cross-reference-24 tags: - TransformOperator diff --git a/docs/build/reference/transformer/index.md b/docs/build/reference/transformer/index.md index 74762b5d3..a9e31201b 100644 --- a/docs/build/reference/transformer/index.md +++ b/docs/build/reference/transformer/index.md @@ -94,8 +94,7 @@ Transform operators transform a one or more sequences of string values to a sequ | [Excel Ln](Excel/Excel_LN.md) | Excel | Excel LN(number): Returns the natural logarithm based on the constant e of the given number. | | [Excel Log](Excel/Excel_LOG.md) | Excel | Excel LOG(number; base): Returns the logarithm of the given number to the specified base. Base is the base for the logarithm calculation. | | [Excel Log10](Excel/Excel_LOG10.md) | Excel | Excel LOG10(number): Returns the base-10 logarithm of the given number. | - - | [Excel map](Replace/excelMap.md) | Replace | Replaces values based on a map of values read from a file in Open XML format (XLSX). The XLSX file may contain several sheets of the form: ```mapFrom,mapTo ,``` and more. An empty string can be created in Excel and alternatives by inserting `=""` in the input line of a cell. If there are multiple values for a single key, all values will be returned for the given key. Note that the mapping table will be cached in memory. If the Excel file is updated (even while transforming), the map will be reloaded within seconds. | + | [Excel map](Replace/excelMap.md) | Replace | Replaces values based on a map of values read from a file in Open XML format (XLSX). The XLSX file may contain several sheets of the form: `mapFrom,mapTo ,` and more. An empty string can be created in Excel and alternatives by inserting `=""` in the input line of a cell. If there are multiple values for a single key, all values will be returned for the given key. Note that the mapping table will be cached in memory. If the Excel file is updated (even while transforming), the map will be reloaded within seconds. | | [Excel Max](Excel/Excel_MAX.md) | Excel | Excel MAX(number_1; number_2; ... number_30): Returns the maximum value in a list of arguments. Number_1; number_2; ... number_30 are numerical values or ranges. | | [Excel Maxa](Excel/Excel_MAXA.md) | Excel | Excel MAXA(value_1; value_2; ... value_30): Returns the maximum value in a list of arguments. Unlike MAX, text can be entered. The value of the text is 0. Value_1; value_2; ... value_30 are values or ranges. | | [Excel Median](Excel/Excel_MEDIAN.md) | Excel | Excel MEDIAN(number_1; number_2; ... number_30): Returns the median of a set of numbers. Number_1; number_2; ... number_30 are values or ranges, which represent a sample. Each number can also be replaced by a reference. | diff --git a/poetry.lock b/poetry.lock index 0ce9d9fef..1df274afd 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1194,6 +1194,131 @@ hyperscan = ["hyperscan (>=0.7)"] optional = ["typing-extensions (>=4)"] re2 = ["google-re2 (>=1.1)"] +[[package]] +name = "pillow" +version = "11.3.0" +description = "Python Imaging Library (Fork)" +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "pillow-11.3.0-cp310-cp310-macosx_10_10_x86_64.whl", hash = "sha256:1b9c17fd4ace828b3003dfd1e30bff24863e0eb59b535e8f80194d9cc7ecf860"}, + {file = "pillow-11.3.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:65dc69160114cdd0ca0f35cb434633c75e8e7fad4cf855177a05bf38678f73ad"}, + {file = "pillow-11.3.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7107195ddc914f656c7fc8e4a5e1c25f32e9236ea3ea860f257b0436011fddd0"}, + {file = "pillow-11.3.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cc3e831b563b3114baac7ec2ee86819eb03caa1a2cef0b481a5675b59c4fe23b"}, + {file = "pillow-11.3.0-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f1f182ebd2303acf8c380a54f615ec883322593320a9b00438eb842c1f37ae50"}, + {file = "pillow-11.3.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4445fa62e15936a028672fd48c4c11a66d641d2c05726c7ec1f8ba6a572036ae"}, + {file = "pillow-11.3.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:71f511f6b3b91dd543282477be45a033e4845a40278fa8dcdbfdb07109bf18f9"}, + {file = "pillow-11.3.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:040a5b691b0713e1f6cbe222e0f4f74cd233421e105850ae3b3c0ceda520f42e"}, + {file = "pillow-11.3.0-cp310-cp310-win32.whl", hash = "sha256:89bd777bc6624fe4115e9fac3352c79ed60f3bb18651420635f26e643e3dd1f6"}, + {file = "pillow-11.3.0-cp310-cp310-win_amd64.whl", hash = "sha256:19d2ff547c75b8e3ff46f4d9ef969a06c30ab2d4263a9e287733aa8b2429ce8f"}, + {file = "pillow-11.3.0-cp310-cp310-win_arm64.whl", hash = "sha256:819931d25e57b513242859ce1876c58c59dc31587847bf74cfe06b2e0cb22d2f"}, + {file = "pillow-11.3.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:1cd110edf822773368b396281a2293aeb91c90a2db00d78ea43e7e861631b722"}, + {file = "pillow-11.3.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:9c412fddd1b77a75aa904615ebaa6001f169b26fd467b4be93aded278266b288"}, + {file = "pillow-11.3.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7d1aa4de119a0ecac0a34a9c8bde33f34022e2e8f99104e47a3ca392fd60e37d"}, + {file = "pillow-11.3.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:91da1d88226663594e3f6b4b8c3c8d85bd504117d043740a8e0ec449087cc494"}, + {file = "pillow-11.3.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:643f189248837533073c405ec2f0bb250ba54598cf80e8c1e043381a60632f58"}, + {file = "pillow-11.3.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:106064daa23a745510dabce1d84f29137a37224831d88eb4ce94bb187b1d7e5f"}, + {file = "pillow-11.3.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:cd8ff254faf15591e724dc7c4ddb6bf4793efcbe13802a4ae3e863cd300b493e"}, + {file = "pillow-11.3.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:932c754c2d51ad2b2271fd01c3d121daaa35e27efae2a616f77bf164bc0b3e94"}, + {file = "pillow-11.3.0-cp311-cp311-win32.whl", hash = "sha256:b4b8f3efc8d530a1544e5962bd6b403d5f7fe8b9e08227c6b255f98ad82b4ba0"}, + {file = "pillow-11.3.0-cp311-cp311-win_amd64.whl", hash = "sha256:1a992e86b0dd7aeb1f053cd506508c0999d710a8f07b4c791c63843fc6a807ac"}, + {file = "pillow-11.3.0-cp311-cp311-win_arm64.whl", hash = "sha256:30807c931ff7c095620fe04448e2c2fc673fcbb1ffe2a7da3fb39613489b1ddd"}, + {file = "pillow-11.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:fdae223722da47b024b867c1ea0be64e0df702c5e0a60e27daad39bf960dd1e4"}, + {file = "pillow-11.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:921bd305b10e82b4d1f5e802b6850677f965d8394203d182f078873851dada69"}, + {file = "pillow-11.3.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb76541cba2f958032d79d143b98a3a6b3ea87f0959bbe256c0b5e416599fd5d"}, + {file = "pillow-11.3.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:67172f2944ebba3d4a7b54f2e95c786a3a50c21b88456329314caaa28cda70f6"}, + {file = "pillow-11.3.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:97f07ed9f56a3b9b5f49d3661dc9607484e85c67e27f3e8be2c7d28ca032fec7"}, + {file = "pillow-11.3.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:676b2815362456b5b3216b4fd5bd89d362100dc6f4945154ff172e206a22c024"}, + {file = "pillow-11.3.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3e184b2f26ff146363dd07bde8b711833d7b0202e27d13540bfe2e35a323a809"}, + {file = "pillow-11.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6be31e3fc9a621e071bc17bb7de63b85cbe0bfae91bb0363c893cbe67247780d"}, + {file = "pillow-11.3.0-cp312-cp312-win32.whl", hash = "sha256:7b161756381f0918e05e7cb8a371fff367e807770f8fe92ecb20d905d0e1c149"}, + {file = "pillow-11.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:a6444696fce635783440b7f7a9fc24b3ad10a9ea3f0ab66c5905be1c19ccf17d"}, + {file = "pillow-11.3.0-cp312-cp312-win_arm64.whl", hash = "sha256:2aceea54f957dd4448264f9bf40875da0415c83eb85f55069d89c0ed436e3542"}, + {file = "pillow-11.3.0-cp313-cp313-ios_13_0_arm64_iphoneos.whl", hash = "sha256:1c627742b539bba4309df89171356fcb3cc5a9178355b2727d1b74a6cf155fbd"}, + {file = "pillow-11.3.0-cp313-cp313-ios_13_0_arm64_iphonesimulator.whl", hash = "sha256:30b7c02f3899d10f13d7a48163c8969e4e653f8b43416d23d13d1bbfdc93b9f8"}, + {file = "pillow-11.3.0-cp313-cp313-ios_13_0_x86_64_iphonesimulator.whl", hash = "sha256:7859a4cc7c9295f5838015d8cc0a9c215b77e43d07a25e460f35cf516df8626f"}, + {file = "pillow-11.3.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ec1ee50470b0d050984394423d96325b744d55c701a439d2bd66089bff963d3c"}, + {file = "pillow-11.3.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:7db51d222548ccfd274e4572fdbf3e810a5e66b00608862f947b163e613b67dd"}, + {file = "pillow-11.3.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2d6fcc902a24ac74495df63faad1884282239265c6839a0a6416d33faedfae7e"}, + {file = "pillow-11.3.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f0f5d8f4a08090c6d6d578351a2b91acf519a54986c055af27e7a93feae6d3f1"}, + {file = "pillow-11.3.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c37d8ba9411d6003bba9e518db0db0c58a680ab9fe5179f040b0463644bc9805"}, + {file = "pillow-11.3.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:13f87d581e71d9189ab21fe0efb5a23e9f28552d5be6979e84001d3b8505abe8"}, + {file = "pillow-11.3.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:023f6d2d11784a465f09fd09a34b150ea4672e85fb3d05931d89f373ab14abb2"}, + {file = "pillow-11.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:45dfc51ac5975b938e9809451c51734124e73b04d0f0ac621649821a63852e7b"}, + {file = "pillow-11.3.0-cp313-cp313-win32.whl", hash = "sha256:a4d336baed65d50d37b88ca5b60c0fa9d81e3a87d4a7930d3880d1624d5b31f3"}, + {file = "pillow-11.3.0-cp313-cp313-win_amd64.whl", hash = "sha256:0bce5c4fd0921f99d2e858dc4d4d64193407e1b99478bc5cacecba2311abde51"}, + {file = "pillow-11.3.0-cp313-cp313-win_arm64.whl", hash = "sha256:1904e1264881f682f02b7f8167935cce37bc97db457f8e7849dc3a6a52b99580"}, + {file = "pillow-11.3.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:4c834a3921375c48ee6b9624061076bc0a32a60b5532b322cc0ea64e639dd50e"}, + {file = "pillow-11.3.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:5e05688ccef30ea69b9317a9ead994b93975104a677a36a8ed8106be9260aa6d"}, + {file = "pillow-11.3.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:1019b04af07fc0163e2810167918cb5add8d74674b6267616021ab558dc98ced"}, + {file = "pillow-11.3.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f944255db153ebb2b19c51fe85dd99ef0ce494123f21b9db4877ffdfc5590c7c"}, + {file = "pillow-11.3.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1f85acb69adf2aaee8b7da124efebbdb959a104db34d3a2cb0f3793dbae422a8"}, + {file = "pillow-11.3.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:05f6ecbeff5005399bb48d198f098a9b4b6bdf27b8487c7f38ca16eeb070cd59"}, + {file = "pillow-11.3.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:a7bc6e6fd0395bc052f16b1a8670859964dbd7003bd0af2ff08342eb6e442cfe"}, + {file = "pillow-11.3.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:83e1b0161c9d148125083a35c1c5a89db5b7054834fd4387499e06552035236c"}, + {file = "pillow-11.3.0-cp313-cp313t-win32.whl", hash = "sha256:2a3117c06b8fb646639dce83694f2f9eac405472713fcb1ae887469c0d4f6788"}, + {file = "pillow-11.3.0-cp313-cp313t-win_amd64.whl", hash = "sha256:857844335c95bea93fb39e0fa2726b4d9d758850b34075a7e3ff4f4fa3aa3b31"}, + {file = "pillow-11.3.0-cp313-cp313t-win_arm64.whl", hash = "sha256:8797edc41f3e8536ae4b10897ee2f637235c94f27404cac7297f7b607dd0716e"}, + {file = "pillow-11.3.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:d9da3df5f9ea2a89b81bb6087177fb1f4d1c7146d583a3fe5c672c0d94e55e12"}, + {file = "pillow-11.3.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:0b275ff9b04df7b640c59ec5a3cb113eefd3795a8df80bac69646ef699c6981a"}, + {file = "pillow-11.3.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:0743841cabd3dba6a83f38a92672cccbd69af56e3e91777b0ee7f4dba4385632"}, + {file = "pillow-11.3.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2465a69cf967b8b49ee1b96d76718cd98c4e925414ead59fdf75cf0fd07df673"}, + {file = "pillow-11.3.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:41742638139424703b4d01665b807c6468e23e699e8e90cffefe291c5832b027"}, + {file = "pillow-11.3.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:93efb0b4de7e340d99057415c749175e24c8864302369e05914682ba642e5d77"}, + {file = "pillow-11.3.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7966e38dcd0fa11ca390aed7c6f20454443581d758242023cf36fcb319b1a874"}, + {file = "pillow-11.3.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:98a9afa7b9007c67ed84c57c9e0ad86a6000da96eaa638e4f8abe5b65ff83f0a"}, + {file = "pillow-11.3.0-cp314-cp314-win32.whl", hash = "sha256:02a723e6bf909e7cea0dac1b0e0310be9d7650cd66222a5f1c571455c0a45214"}, + {file = "pillow-11.3.0-cp314-cp314-win_amd64.whl", hash = "sha256:a418486160228f64dd9e9efcd132679b7a02a5f22c982c78b6fc7dab3fefb635"}, + {file = "pillow-11.3.0-cp314-cp314-win_arm64.whl", hash = "sha256:155658efb5e044669c08896c0c44231c5e9abcaadbc5cd3648df2f7c0b96b9a6"}, + {file = "pillow-11.3.0-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:59a03cdf019efbfeeed910bf79c7c93255c3d54bc45898ac2a4140071b02b4ae"}, + {file = "pillow-11.3.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f8a5827f84d973d8636e9dc5764af4f0cf2318d26744b3d902931701b0d46653"}, + {file = "pillow-11.3.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ee92f2fd10f4adc4b43d07ec5e779932b4eb3dbfbc34790ada5a6669bc095aa6"}, + {file = "pillow-11.3.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c96d333dcf42d01f47b37e0979b6bd73ec91eae18614864622d9b87bbd5bbf36"}, + {file = "pillow-11.3.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4c96f993ab8c98460cd0c001447bff6194403e8b1d7e149ade5f00594918128b"}, + {file = "pillow-11.3.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:41342b64afeba938edb034d122b2dda5db2139b9a4af999729ba8818e0056477"}, + {file = "pillow-11.3.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:068d9c39a2d1b358eb9f245ce7ab1b5c3246c7c8c7d9ba58cfa5b43146c06e50"}, + {file = "pillow-11.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:a1bc6ba083b145187f648b667e05a2534ecc4b9f2784c2cbe3089e44868f2b9b"}, + {file = "pillow-11.3.0-cp314-cp314t-win32.whl", hash = "sha256:118ca10c0d60b06d006be10a501fd6bbdfef559251ed31b794668ed569c87e12"}, + {file = "pillow-11.3.0-cp314-cp314t-win_amd64.whl", hash = "sha256:8924748b688aa210d79883357d102cd64690e56b923a186f35a82cbc10f997db"}, + {file = "pillow-11.3.0-cp314-cp314t-win_arm64.whl", hash = "sha256:79ea0d14d3ebad43ec77ad5272e6ff9bba5b679ef73375ea760261207fa8e0aa"}, + {file = "pillow-11.3.0-cp39-cp39-macosx_10_10_x86_64.whl", hash = "sha256:48d254f8a4c776de343051023eb61ffe818299eeac478da55227d96e241de53f"}, + {file = "pillow-11.3.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:7aee118e30a4cf54fdd873bd3a29de51e29105ab11f9aad8c32123f58c8f8081"}, + {file = "pillow-11.3.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:23cff760a9049c502721bdb743a7cb3e03365fafcdfc2ef9784610714166e5a4"}, + {file = "pillow-11.3.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6359a3bc43f57d5b375d1ad54a0074318a0844d11b76abccf478c37c986d3cfc"}, + {file = "pillow-11.3.0-cp39-cp39-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:092c80c76635f5ecb10f3f83d76716165c96f5229addbd1ec2bdbbda7d496e06"}, + {file = "pillow-11.3.0-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:cadc9e0ea0a2431124cde7e1697106471fc4c1da01530e679b2391c37d3fbb3a"}, + {file = "pillow-11.3.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:6a418691000f2a418c9135a7cf0d797c1bb7d9a485e61fe8e7722845b95ef978"}, + {file = "pillow-11.3.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:97afb3a00b65cc0804d1c7abddbf090a81eaac02768af58cbdcaaa0a931e0b6d"}, + {file = "pillow-11.3.0-cp39-cp39-win32.whl", hash = "sha256:ea944117a7974ae78059fcc1800e5d3295172bb97035c0c1d9345fca1419da71"}, + {file = "pillow-11.3.0-cp39-cp39-win_amd64.whl", hash = "sha256:e5c5858ad8ec655450a7c7df532e9842cf8df7cc349df7225c60d5d348c8aada"}, + {file = "pillow-11.3.0-cp39-cp39-win_arm64.whl", hash = "sha256:6abdbfd3aea42be05702a8dd98832329c167ee84400a1d1f61ab11437f1717eb"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:3cee80663f29e3843b68199b9d6f4f54bd1d4a6b59bdd91bceefc51238bcb967"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:b5f56c3f344f2ccaf0dd875d3e180f631dc60a51b314295a3e681fe8cf851fbe"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e67d793d180c9df62f1f40aee3accca4829d3794c95098887edc18af4b8b780c"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:d000f46e2917c705e9fb93a3606ee4a819d1e3aa7a9b442f6444f07e77cf5e25"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:527b37216b6ac3a12d7838dc3bd75208ec57c1c6d11ef01902266a5a0c14fc27"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:be5463ac478b623b9dd3937afd7fb7ab3d79dd290a28e2b6df292dc75063eb8a"}, + {file = "pillow-11.3.0-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:8dc70ca24c110503e16918a658b869019126ecfe03109b754c402daff12b3d9f"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:7c8ec7a017ad1bd562f93dbd8505763e688d388cde6e4a010ae1486916e713e6"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:9ab6ae226de48019caa8074894544af5b53a117ccb9d3b3dcb2871464c829438"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fe27fb049cdcca11f11a7bfda64043c37b30e6b91f10cb5bab275806c32f6ab3"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:465b9e8844e3c3519a983d58b80be3f668e2a7a5db97f2784e7079fbc9f9822c"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5418b53c0d59b3824d05e029669efa023bbef0f3e92e75ec8428f3799487f361"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:504b6f59505f08ae014f724b6207ff6222662aab5cc9542577fb084ed0676ac7"}, + {file = "pillow-11.3.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:c84d689db21a1c397d001aa08241044aa2069e7587b398c8cc63020390b1c1b8"}, + {file = "pillow-11.3.0.tar.gz", hash = "sha256:3828ee7586cd0b2091b6209e5ad53e20d0649bbe87164a459d0676e035e8f523"}, +] + +[package.extras] +docs = ["furo", "olefile", "sphinx (>=8.2)", "sphinx-autobuild", "sphinx-copybutton", "sphinx-inline-tabs", "sphinxext-opengraph"] +fpx = ["olefile"] +mic = ["olefile"] +test-arrow = ["pyarrow"] +tests = ["check-manifest", "coverage (>=7.4.2)", "defusedxml", "markdown2", "olefile", "packaging", "pyroma", "pytest", "pytest-cov", "pytest-timeout", "pytest-xdist", "trove-classifiers (>=2024.10.12)"] +typing = ["typing-extensions ; python_version < \"3.10\""] +xmp = ["defusedxml"] + [[package]] name = "platformdirs" version = "4.11.8" @@ -2127,4 +2252,4 @@ tomli = ">=2.4.0" [metadata] lock-version = "2.1" python-versions = "^3.11" -content-hash = "d80d76a1d975fc810e59f1f3dd32acb40d17f06539f1c9ac2bfa8ac09923c8cb" +content-hash = "11d58954fa95ccfda69482e9b9f98c6b353b358999f725966da0fa5b9459f9dc" diff --git a/pyproject.toml b/pyproject.toml index 6ec3e1310..efd06901b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -40,6 +40,7 @@ jinja2 = "^3.1.6" # release. This narrowness is intentional -- bump it manually and re-run # `task build` plus `tools/check_zensical_output.py` when doing so. zensical = "^0.0.62" +pillow = "^11.3.0" [tool.poetry.group.dev.dependencies] linkcheckmd = "^1.4.0" diff --git a/tasks/backlog.md b/tasks/backlog.md index 5dea765fe..a0e658bb7 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -1,270 +1,360 @@ -# Backlog: temporary tag-listing renderer +# Backlog: print-on-demand book block -Work breakdown for [spec.md](spec.md). **Removed 2026-09-03** - see spec.md's Removal -section; superseded by Zensical 0.0.58's native `tags` listings. +Work breakdown for [spec.md](spec.md). **Status 2026-09-15: P0-P12 done and verified, P13-P17 open.** -**B0-B8: done** (2026-08-23). Listings render on `/tags/` and `/tutorials/`, guarded by -three required checks, 30 unit tests, `task preview` added for accurate previews. +The previous content of this file (the temporary tag-listing renderer) is in the git history. -**C1-C4: done** (2026-08-23). Every page tag chip links to its section on `/tags/` - -703 chips across 531 pages - guarded by two more required checks and 10 more unit tests. -Q6 settled as recommended. +Every task names how it is verified. A task is not done until that verification passes. --- -## B0 — Decide the open questions +## P0 - Decisions - done -Spec §5. My recommendations, all "match production / fail loudly": +Spec §4, D1-D11: BoD, A4, black and white on 80 g, no ISBN, GitHub IDs as authors, separate screen and +print editions, section modes with A.3 and Release Notes as lists, page references and URL footnotes, +no logo or version on text pages, 10 pt body, optional Ghostscript normalization. -| # | Question | Recommendation | -|---|---|---| -| 1 | Icons for the 14 unmapped tags | Render plain, as production does | -| 2 | `Graph-Insights` vs `GraphInsights` duplicate | Out of scope; separate content fix | -| 3 | `Load Balancer` mapping missing from `HEAD` | Restore the two `mkdocs.yml` lines | -| 4 | Marker present but unrenderable | Fail the build | -| 5 | `/tags/` size (45 sections, 531 refs) | Include everything | +--- + +## P1 - Print edition - **done** + +`task pdf:print` builds `dist/documentation-eccenca-com--print.pdf` next to the screen PDF: +the same merge and pandoc run, Typst compiled with `--input edition=print`. `style.typ` reads the input +once and branches where the editions differ. `task pdf` stays as it is. + +**Verify:** both PDFs build; the screen PDF still has 1680 pages and an unchanged page 4. +**Est:** small. **Depends on:** nothing. -**Output:** decisions recorded in the spec. **Est:** one review pass. +**Done 2026-09-15:** `dec-tool build-pdf --edition print` (`PDF_EDITION`), intermediates in +`dist/pdf/print/`, flag `print-edition` in `style.typ`. Against a baseline built from `c20d74b94` +before any change: the screen PDF has 1680 pages, the text of all pages is identical and page 4 differs +by 0 pixels. The print edition builds (1585 pages at that point). --- -## B1 — Tag index builder +## P2 - Mirrored page geometry - **done** -Walk `docs/**/*.md`, parse front matter, build `{tag: [(title, src_path)]}`. +A4, `binding: left`, margins as `inside`/`outside` in the print edition; the peach bands' outset +mirrors with them. -- title = front-matter `title:`, else first body `# ` heading, else skip with a warning -- skip pages with no `tags:` -- tolerate malformed YAML without crashing the build -- pure function over `docs/`, no `site/` knowledge — keeps it unit-testable +**Verify:** render a spread (an even and the following odd page): the text blocks mirror, and the +text width is unchanged. +**Est:** small. **Depends on:** P1. -**Verify:** 531 tagged pages, 45 distinct tags, `TransformOperator` = 237. -**Est:** small. **Depends on:** nothing. +**Done 2026-09-15:** inside 3.0 cm, outside 2.0 cm. The header of the print edition is empty (P3), so +its top margin is 2.5 cm instead of 3.9 cm; the bands' outset is symmetric and needs no mirroring. +Measured with `pdftotext -bbox` on pages 20 and 21: even page margins 2.00 cm left and 3.00 cm right, +odd page 3.00 cm left and 2.00 cm right, text width 16.00 cm on both. --- -## B2 — Marker parser +## P3 - Running titles and page numbers - **done** -Recognise both forms in built HTML and extract the filter: - -``` - -> no filter - -> include=[BeginnersTutorial] -``` +- verso: page number at the outer left, part title; recto: page title, page number at the outer right +- no `| total`, no logo, no version stamp on text pages +- none on the title page, imprint, part covers and blank pages -The argument is YAML-ish but not valid YAML (unquoted `[X]` inside braces parses fine, but -do not assume). Parse defensively; an unrecognised argument is an error, not a silent -no-filter. +**Verify:** a script reads `pdftotext -bbox` and asserts, for every page with a number, x below 20 % +of the page width on even pages and above 80 % on odd pages. Render two spreads. +**Est:** medium - the footer queries already exist; the parity and the exclusions are new. +**Depends on:** P2. -**Verify:** finds exactly 4 markers across 2 files in the current build. -**Est:** small. **Depends on:** nothing. +**Done 2026-09-15:** `print-footer()` and `bare-page()` in `style.typ`; the print header is empty. The +check over all 1604 pages of the print edition: 1575 pages carry a footer and each has its page number +at the outer edge (left on even, right on odd pages); the 29 pages without one are exactly the title +page, the imprint, the 9 part covers and the 18 blank pages. The checker becomes part of the preflight +report (P14). --- -## B3 — HTML renderer +## P4 - Recto starts and blank pages - **done** -Emit the markup in spec §3 for one listing. +- title page 1, imprint 2, front contents 3 +- part cover, part contents and part text each start recto +- a blank page has no furniture: a state set by the page break, read by header and footer +- total padded to even -- anchor id `tag:` + lowercase, spaces → hyphens -- chip class from `extra.tags`; bare `md-tag` when unmapped (pending Q1) -- relative href from the listing page to each target -- escape titles -- ordering per spec §3 +**Verify:** every part cover and the front contents on an odd page; every blank page has no text in +`pdftotext`; page count even. +**Est:** medium - suppressing furniture on inserted blank pages is the fiddly part (spec §1). +**Depends on:** P3. -**Verify:** byte-compare one rendered block against the production sample in the spec. -**Est:** medium — the relative-URL computation is the fiddly part. -**Depends on:** B1, B2, Q1. +**Done 2026-09-15:** `recto-break()` brackets `pagebreak(weak: true, to: "odd")` with two metadata +markers, and a page strictly between them counts as blank. Typst cannot pad to an even count itself - +a page break that depends on the page count never converges - so the build reads the unpadded count +with `typst eval` (`unpadded_pages`) and compiles with `--input pad=true` when it is odd (1603 → 1604). +Checked on all pages: title page 1, imprint 2, front contents 3; the 9 part covers and the 9 part +contents start on odd pages; 18 blank pages, all even, all without text, never three in a row. --- -## B4 — `tools/render_tag_listings.py` +## P5 - Title page - **done** -Wire B1–B3 into a CLI matching `localize_bundle_assets.py`'s shape: takes `[site_dir]`, -prints `[OK]` lines per marker, exits non-zero with a problem list. +Publisher **eccenca GmbH** on the title block; site link and copyright move to the imprint. -- reads `mkdocs.yml` for `extra.tags` and the two `listings_*_sort_by` settings -- idempotent: re-running on an already-rendered `site/` is a no-op, not an error - (`localize_bundle_assets.py` needed this and it was easy to get wrong) -- fails if any marker remains after processing +**Verify:** render page 1. +**Est:** trivial. **Depends on:** P4. -**Verify:** `python tools/render_tag_listings.py site` twice in a row, second run clean. -**Est:** small once B1–B3 exist. **Depends on:** B1, B2, B3, Q4. +**Done 2026-09-15:** rendered page 1: logo, the house title block and version, the publisher from +`tools/pdf/print.yml` at the foot of the page; no date, link or copyright. The screen title page is +unchanged. --- -## B5 — Wire into the build +## P6 - Author list - **done** + +`dec-tool pdf-authors` (run by `task pdf:authors`) writes `tools/pdf/authors.yml`: GitHub ID and +commits, most commits first, ties by ID case-insensitive. + +- source: the GitHub contributors API without anonymous entries +- excludes accounts of type `Bot` and agent IDs matching `claude` or `codex` +- committed, so the PDF build stays offline and reproducible and a changed list shows in review -Add to `build` in `Taskfile.yml`, after `zensical build --strict` and alongside -`localize_bundle_assets.py`. Order relative to the localizer does not matter — they touch -disjoint files — but keep the localizer first so the more security-relevant step runs -regardless. +**Verify:** unit tests for ordering, tie-break and exclusion; the generated file matches the spec §3 +table (22 IDs). +**Est:** small. **Depends on:** nothing. -**Verify:** `task clean build` renders listings; `task check` still passes. -**Est:** trivial. **Depends on:** B4. +**Done 2026-09-15:** `tools/pdf_authors.py`, `tests/test_pdf_authors.py` (4 tests: order and tie-break, +exclusion of bots, agents and anonymous entries, pagination, file format), `task pdf:authors`. The +generated `tools/pdf/authors.yml` lists the 22 IDs of spec §3 in the same order and passes yamllint - +list items are indented, which PyYAML's default dumper does not do. --- -## B6 — Promote the guard +## P7 - Imprint - **done** + +Page 2 as spec R3, with the publisher address `eccenca GmbH, Hainstraße 8, 04109 Leipzig, Germany`, no +ISBN. Typst reads `tools/pdf/authors.yml` with `yaml()`; edition, commit and date come from the +existing `--input` values. -In `check_zensical_output.py`, `tag-listings` currently sits in PENDING and reports -unexpanded markers. Once we render them ourselves it becomes a feature we own, so it -belongs in REQUIRED — same reasoning as redirects and comments. +**Verify:** render page 2; the author order matches `authors.yml`. +**Est:** small. **Depends on:** P5, P6. -Keep a separate PENDING probe for "Zensical started doing this itself", so the removal -signal in spec §6 still fires. Distinguishing the two is the fiddly bit: our own output and -Zensical's would both look like a populated listing. Suggest keying the PENDING probe on a -marker being *already expanded before* our script runs. +**Done 2026-09-15:** `imprint()` in `style.typ`, set at the foot of page 2: edition stamp with commit, +publisher and address from `tools/pdf/print.yml`, the 22 author IDs, licence and copyright, the online +edition - with a sentence that the print edition shortens sections whenever `print.yml` lists one as +`list` or `omit` - and the typesetting. Checked on the rendered page and in its text: the author IDs match +`authors.yml` in order. The licence URL is set as a string, because Typst links URLs written in markup. + +--- -**Verify:** deliberately skip the render step; `task check` must fail. -**Est:** medium — mostly deciding the probe. **Depends on:** B4, B5. +## P8 - Section modes - **done** + +`tools/pdf/print.yml` maps `nav.yml` section paths to `full`, `list` or `omit`; `tools/build_pdf.py` +reads it for the print edition only. + +- `omit`: the section's pages are not merged; the part contents name the online edition instead +- `list` for the operator reference (`build/reference/`): merge the section page and the five overview + pages, drop their children; links from the overview tables to dropped pages print as plain text +- `list` for Release Notes (`release-notes/`): replace the release pages with one generated table, + Release | Summary, the summary being the release page's first paragraph or, without one, its component + headings +- a section in `list` or `omit` mode starts with a sentence naming its online URL +- an unknown path or mode in `print.yml` fails the build + +**Verify:** unit tests per mode on small fixtures; with the default configuration A.3 takes about +17 pages and Release Notes about 3; switching either to `full` restores today's pages; no internal +link targets a dropped page. +**Est:** medium - the release summary fallback and the links into dropped pages are the fiddly parts. +**Depends on:** P1. + +**Done 2026-09-15:** `load_section_rules`, `apply_section_rules` (`list_section`, `omit_section`) and +`render_generated` in `tools/build_pdf.py`; `tests/test_build_pdf_print.py` (10 tests). A `list` section +keeps its own page and its subsections' overview pages; a page no overview lists goes into a table under +its heading, one per run of pages. Release Notes have no overview pages, so they become 8 tables - one +per year - with the 22 releases. Measured on the print edition: A.3 takes 17 pages (pp. 23-39, 389 pages +dropped), the Release Notes part 6 pages including cover, contents and blank pages; the book goes from +1604 to 1012 pages. Setting both sections back to `full` gives 1604 pages again. The two notes name +`…/26.2/build/reference/` and, since `release-notes/` has no page, the first release page. Links from +the overview tables to dropped pages print as text; 105 links from other pages to dropped operator +pages lead to the published site - a dropped page has no label, so no internal link can target one. --- -## B7 — Tests +## P9 - Links on paper - **done** -`tests/test_render_tag_listings.py`, following `tests/test_update_di_reference.py`. +In the print edition: internal links print their text plus `(p. N)`; external links print their text +with a footnote holding the URL; no colour, underline, arrow or link annotation. -- title resolution: front matter wins over H1; H1 fallback; neither → warn -- slugification: `Load Balancer` → `tag:load-balancer` -- sort orders: casefold for tags, title for items, marker order for `include:` -- unmapped tag → no `md-tag-icon` class -- marker parsing: both forms, plus a malformed one -- relative hrefs from different depths +**Verify:** sample pages; the number of footnotes equals the number of external links in the printed +sections; no page reference is `p. 0`; `strings` finds no `/Annots` in the book block. +**Est:** medium - thousands of page lookups; watch for Typst's "layout did not converge" warning. +**Depends on:** P1, P8. -**Verify:** `task test:unit` stays green. -**Est:** medium. **Depends on:** B1–B4. +**Done 2026-09-15:** the print branch of `show link` in `style.typ`; contents entries are laid out +without their link. Of 524 external links, 73 print their address as their own text and get no +footnote; 451 get one, 2 of them repeating the address their text already shows. 2154 page references, +none `p. 0`. A reference printed only for a target on another page never let the layout converge - it +moves lines, which moves the target back - so it is printed always, and the build now has no +convergence warning. Deviation from the check above: Typst's own footnotes link marker and entry, so +the book block keeps 902 internal link annotations; none of them leads to an address (`/S /URI`: 0). +The optional Ghostscript pass (P13) removes them. --- -## B8 — Documentation +## P10 - Typography for print - **done** -Extend the README section added during the migration. It currently lists tag listings among -the three missing features; that becomes "reimplemented locally, tracked for removal", -leaving social cards and revision dates as the genuinely-missing two. +- paragraph spacing 1.2 em and block spacing 1.0 em (measured −6.5 % pages) +- hyphenation on, widow and orphan costs +- a 6 pt floor for shrinking terminal tables +- measure Regular instead of Light for the body; decide with a greyscale print sample +- body stays 10 pt; 9 or 8 pt only as a fallback if the preflight page limit is exceeded (D10) -**Est:** trivial. **Depends on:** B5. +**Verify:** page count after each change; render pages with justified text. +**Est:** small. **Depends on:** P1. ---- +**Done 2026-09-15:** paragraph spacing 1.2 em and block spacing 1.0 em take the book from 1012 to 962 +pages; hyphenation on for body text, off for titles; code in a shrinking terminal table never below +6 pt. Typst's defaults already cost widows and orphans at 100 % (`text.costs`, checked), so they needed +no setting. A Regular body costs 2 pages (963 instead of 961 before padding); `body-weight` in +`tools/pdf/print.yml` switches it, and it stays `light` until a printed sample decides. Sample pages 26 +and 268 checked on the render: justified text without gaps, code and terminal table legible. -## Sequencing +--- -``` -B0 ─┬─> B1 ─┬─> B3 ──> B4 ──> B5 ──> B6 ──> B8 - └─> B2 ─┘ └──> B7 -``` +## P11 - Black-and-white palette - **done** -B1 and B2 are independent and can start as soon as Q1/Q4 are settled. +- grey areas at least 20 % black (BoD): `ec-wash` code and note grounds, peach bands - raise to 20 % or + replace fills with rules +- print-edition values for orange, link blue and the admonition accents that stay distinguishable in + grey +- check the red frames of the four part diagrams -## Risks +**Verify:** render sample pages with the greyscale preview of P13 and compare with the screen +edition; the preflight measures the lightest fill. +**Est:** small. **Depends on:** P1. -- **Relative URL computation** is where this most likely breaks — mike serves the site under - `/latest/` and `/26.2/`, so anything absolute fails silently in one context. Acceptance - requires resolving every generated link against `site/` on disk. -- **`/tags/` is large.** 531 references in one page. Watch build time; if it becomes - noticeable, that is an argument for revisiting Q5. -- **Divergence from production markup.** We match it today, but a Material update could - change the markup and this becomes a slow drift. Mitigated by the whole thing being - temporary and by the spec pinning a production sample. -- **The removal signal is the weak point** (B6). If it never fires, this "temporary" script - becomes permanent. Worth a calendar reminder to re-check backlog #38 rather than relying - only on the probe. +**Done 2026-09-15:** in print, text and alarm accents are black, the orange a dark grey, the peach bands +20 % black; code, code spans and admonitions have no ground - a thin frame marks a code block, the bar on +the left an admonition. The greyscale preview of P13 does not exist yet, so Ghostscript's `pnggray` +rendered all 962 pages instead: on the 610 pages without images no area is lighter than 20 % black, and +the bands measure 204 (20 % black) on 36 pages. The part diagrams' red frames are part of their images and +stay visible as a dark frame on the covers. --- -# Part 2: linked tag chips (spec §9) +## P12 - Print images - **done** -Rendering the listings created 45 anchors that nothing points at. Material links every -per-page tag chip to its section on `/tags/`; Zensical emits inert ``s. Measured: -**531 pages, 703 chips, 0 links.** +In `tools/build_pdf.py`, for the print edition: every image Typst embeds is a normalized copy in +`dist/pdf/images/`. -Smaller than Part 1 and a different mechanism - a **template override**, not another -post-build pass, so it also works under `task serve`. +- alpha composited onto white, so the book block carries no transparency +- resampled (Lanczos) to 300 ppi at its printed width: text width (16 cm) when the image is wider than + the column, its natural size otherwise +- no longer scaled up beyond its natural size (spec §6) +- originals under `site/` untouched; cached by content hash, so a rebuild does not resample again -## Q6 — Decide before coding — **done**, all as recommended +Needs an image library. Pillow is importable today only as a transitive dependency (11.3.0) and is not +declared in `pyproject.toml` - declare it before relying on it. ImageMagick is the alternative: installed +locally, not yet in CI. -Spec §9 open questions: +**Verify:** unit tests on fixture PNGs (alpha, small, large); in the book block `pdfimages -list` shows +no soft mask and no image below 300 ppi; render pages with former low-resolution screenshots. +**Est:** medium - the printed-width rule must match how Typst sizes images. **Depends on:** P1. -| # | Question | Recommendation | -|---|---|---| -| 1 | Slug computed twice (MiniJinja + Python) | Accept the duplication, but assert reachability in the check | -| 2 | Tag whose only page is the listing page itself → dead anchor | Covered by the same assert; no special case | -| 3 | `hide: tags` pages | Keep the existing template branch; no decision | +**Done 2026-09-15:** `print_image` and `printed_width_pt` in `tools/build_pdf.py`, Pillow declared in +`pyproject.toml` (it was installed only as a leftover of CairoSVG); `tests/test_build_pdf_images.py` +(7 tests). The printed width follows how Typst sizes an image, measured in a probe: a percentage of the +16 cm column, otherwise pixels × 72 / declared dpi (72 without one), never wider than the column and +never scaled up - so "no longer scaled up" needed no change. The copy declares 300 dpi and keeps that +printed width. In the book block: 586 images, 0 soft masks, the lowest at 300 ppi, none below; 571 +copies in `dist/pdf/print/images/`. The PDF grows from 72 to 165 MB with the lossless upsampled copies. -**Est:** one review pass. +--- -Decision: go with all above recommendations! +## P13 - Normalization pass (optional) ---- +`dec-tool pdf-normalize ` (run by `task pdf:print` with `--normalize` / `PDF_NORMALIZE=1`), spec §7: -## C1 — `overrides/partials/tags.html` — **done** +- Ghostscript pdfwrite with `-dPDFX=4`, CMYK conversion, bicubic downsampling of colour and grey + images above 300 dpi, Flate re-encoding, `-dPreserveAnnots=false` +- prefix file `tools/pdf/PDFX_def.ps`, derived from Ghostscript's `lib/PDFX_def.ps`, with title and + a FOGRA39 output intent; the profile passed with `--permit-file-read` +- ISO Coated v2 (FOGRA39) profile vendored under `tools/pdf/icc/` with its ECI licence, listed in the + README asset table - or downloaded in CI if the licence does not allow vendoring +- a `--gray` preview variant for P11 +- fails when the output lacks `GTS_PDFXVersion (PDF/X-4)` - Ghostscript silently falls back to plain + PDF when a page still carries an annotation +- never `-sOutputICCProfile` together with `-dPDFX` (crashes, truncated file) -Copy Zensical's `partials/tags.html` verbatim and replace only the `tag.url` branch: +**Verify:** on the full book block: PDF/X-4 marker in info and XMP, one output intent naming FOGRA39, +`pdfimages -list` shows only `cmyk` images and none above 300 ppi, `pdffonts` all embedded, page count +unchanged; render sample pages before and after and compare. +**Est:** small - the recipe is verified on 38 pages. **Depends on:** P9 (no annotations), P12. -```jinja -{% set anchor = "tags/" | url ~ "#tag:" ~ (tag.name | lower | replace(" ", "-")) %} -{{- tag.name -}} -``` +--- -Keep everything else byte-identical, as `tabs-item.html` does, so it can be re-synced. -Retain the `hide: tags` branch and the `md-tag-shadow` / `md-tag--` class logic. +## P14 - Preflight report -**Done:** output is byte-identical to production's, e.g. -`Configuration`. -The override differs from Zensical's stock template by exactly the one branch. +`dec-tool pdf-preflight `: A4 page size, page count at most 1,200 and even, fonts embedded, images +below 300 ppi, soft masks, lightest fill below 20 % black, page-number position (P3's check), blank +pages blank. Non-zero exit on a violation; run by `task pdf:print`, on the normalized file when P13 ran. ---- +**Verify:** unit tests on small fixture PDFs; deliberately break one rule, the report must fail. +**Est:** medium. **Depends on:** P3, P4, P8. -## C2 — Assert every chip anchor resolves — **done** +--- -New required check in `check_zensical_output.py`: for each `href="…/tags/#tag:X"` in the -output, `#tag:X` must exist on `/tags/`. This is what keeps the MiniJinja slug and -`render_tag_listings.tag_slug()` from silently drifting apart. +## P15 - Low-resolution originals -Cheap: collect the anchor ids from `/tags/` once, then set-compare against the hrefs. +49 images are below 150 ppi at printed size; resampling (P12) hides that from the preflight but adds no +detail. Work through the list: replace with a fresh screenshot, or accept in spec §8. -**Done:** `tag-chips-linked` (703 chips on 531 pages) and `tag-chips-resolve` (every -anchor exists), both required. Verified by removing `replace(" ", "-")` from the template: -the check failed on `#tag:load balancer` and `#tag:application view`, the only two -multi-word tags. +**Verify:** every entry replaced or accepted. +**Est:** medium, mostly content. **Depends on:** P12. --- -## C3 — Tests — **done** +## P16 - CI -Extend `tests/test_render_tag_listings.py`, or a sibling, with the slug-parity case: the -Python `tag_slug()` and the template's expression must agree for every tag in use - -including `Load Balancer` (space) and `Graph-Insights` (existing hyphen). +`.github/workflows/pdf.yml` also builds the normalized book block and uploads it with the preflight +report, as a separate artifact. Installs Ghostscript on the runner. -Template rendering itself is covered by C2 against the real build rather than by unit test; -MiniJinja is not worth mocking for three filters. - -**Done:** 10 tests reading the override and asserting its slug expression matches -`tag_slug()`. 59 tests pass in total. +**Verify:** a push to `main` produces both artifacts. +**Est:** small. **Depends on:** P1, P13, P14. --- -## C4 — Documentation — **done** +## P17 - Cover (separate deliverable) -README: the "reimplemented here" table gains a row for the chip links, pointing at the same -`#38` and the same removal trigger. Spec §7 already lists the override for deletion. +Front, spine and back; the spine width follows from the final page count and 80 g paper. Tracked here +so it is not forgotten; it has its own spec. -**Done.** +**Depends on:** P8, P14. --- ## Sequencing +```text +P1 ──> P2 ──> P3 ──> P4 ──> P5 ──> P7 + │ │ P6 ──┘ + ├──> P8 ──> P9 ──────┼─────────────┐ + ├──> P10 │ │ + ├──> P11 │ │ + └──> P12 ──> P15 │ │ + └──────────────┼──> P13 <────┘ + └──> P14 ──> P16 +P17 after P8 and P14 ``` -Q6 ──> C1 ──> C2 ──> C3 - └──> C4 -``` + +P1, P6, P8, P10, P11 and P12 can start now. ## Risks -- **Slug drift** between the template and the Python renderer is the only real one, and C2 - exists specifically to make it loud. Do not skip C2 to save time. -- **Chips link to `/tags/`, which is a large page.** 45 sections, 531 references. Jumping - to an anchor there is fine, but it is a heavy page to load from a chip click. Matches - production, so not a regression - noting it because it is the sort of thing that gets - raised later as if it were new. -- **Override drift** if Zensical changes its `tags.html`. Same exposure as - `tabs-item.html`; mitigated by keeping the copy byte-identical apart from the one branch. +- **The page budget is an estimate.** It adds measured savings that were taken separately; P8 and P10 + must re-measure, and P14 enforces the limit. +- **Ghostscript falls back to plain PDF silently.** One annotation left anywhere and the output is not + PDF/X; P13 checks the marker instead of trusting the exit code. +- **CMYK conversion and downsampling change screenshots.** Compare renders before and after P13 on pages + with dense UI text; Flate keeps the re-encoding lossless, the bicubic resampling does not. +- **Upsampled images look sharper in the preflight than on paper.** P15 exists because P12 cannot add + detail that is not there. +- **Black and white loses colour cues** in screenshots and in the part diagrams' red frames, and today's + light fills fall below BoD's 20 % rule; only a greyscale sample shows how much. +- **Page references cost compile passes.** Thousands of `(p. N)` lookups can move page breaks that move + page numbers; Typst stops after five layout attempts and warns. +- **80 g paper is for publishers only at BoD.** Without a publisher account the book prints on 90 g, + where the limit is 1,050 pages. +- **The screen PDF must not drift.** Every print change goes behind the `edition=print` switch (P1), and + the screen PDF's page count is part of every verification. diff --git a/tasks/spec.md b/tasks/spec.md index ad9359c9e..843af395d 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -1,267 +1,371 @@ -# Spec: temporary tag-listing renderer +# Spec: print-on-demand book block -**Status:** accepted 2026-08-23, fully implemented §1-§9, **removed 2026-09-03**. -Zensical 0.0.58 ships native `tags` listings and populates `tag.url` itself; the -"Removal" steps below were carried out as part of the 0.0.57 -> 0.0.58 bump. -**Replaced:** Material's `tags` plugin listings, which Zensical did not implement -([zensical/backlog#38](https://github.com/zensical/backlog/issues/38)). -**Lifetime:** delete the moment Zensical ships listings. See "Removal" below. +**Status:** accepted 2026-09-14 - all decisions made (§4). Backlog P0-P12 implemented 2026-09-15 +(`task pdf:print`, 962 pages); P13-P17 open. The backlog's "Done" notes record where the implementation +refines this spec. +**Branch:** `feature/print-on-demand`, based on `main` at `c20d74b94` (PDF export merged). +**Goal:** a *book block* - the interior file of a printed, perfect-bound book - built next to the +screen PDF, which BoD accepts without rework. +**Not in scope:** the cover file (front, spine, back); see §9. ---- - -## 1. Problem - -`tools/check_zensical_output.py` reports this every build: - -``` -[PEND] tag-listings: /tags/ article has 3 words, 4 unexpanded - marker(s) on 2 page(s) (backlog #38) -``` - -Per-page tag *chips* work — Zensical renders them, and the icon CSS with them. What is -missing is the *listings*: the generated index of which pages carry which tag. - -Two pages are affected, carrying four markers between them: - -| Page | Marker | Renders today | -|---|---|---| -| `docs/tags.md` | `` | nothing — page body is just its title | -| `docs/tutorials/index.md` | `` | nothing | -| `docs/tutorials/index.md` | `` | nothing | -| `docs/tutorials/index.md` | `` | nothing | - -`/tutorials/` is the more visible loss: it is a landing page whose entire purpose is the -generated list, and it currently shows an intro paragraph followed by blank space. - -## 2. Approach - -**Post-build HTML injection**, mirroring the existing `tools/localize_bundle_assets.py`. - -Zensical passes the markers through to the output verbatim as HTML comments: - -```html - - -``` - -so they are addressable in `site/**/*.html` after the build. A script walks the corpus for -tag front matter, renders the listing HTML, and substitutes it for each marker. - -### Why this shape +The previous content of this file (the temporary tag-listing renderer, removed 2026-09-03) is in the +git history. -- **Sources stay untouched.** The pages keep Material's own marker syntax, so when - Zensical implements listings they light up natively and this script is deleted. No - migration back. -- **Precedent in-tree.** `localize_bundle_assets.py` already post-processes `site/` inside - `task build`, with the same "assert loudly if the expected pattern is missing" contract. -- **No new dependencies, no template overrides.** Icon styling already works (below). - -### Alternatives rejected - -| Option | Why not | -|---|---| -| Pre-build: expand markers into `docs/*.md` | Mutates tracked sources; dirty tree after every build | -| Generate a partial + `--8<--` snippets include | Requires editing both pages away from Material syntax, then back later; needs two-phase build | -| Override a Zensical template | Zensical has no listings template to override — the feature is absent, not broken | -| Write a Zensical plugin | Zensical has no plugin API | - -## 3. Output contract - -Reproduce production's markup exactly. Sample from -`https://documentation.eccenca.com/latest/tutorials/`: - -```html -

-BeginnersTutorial¤

- -``` - -### Rules, all verified against production +--- -**Anchor id** — `tag:` + tag lowercased with spaces replaced by hyphens. -`Load Balancer` → `tag:load-balancer`, `Graph-Insights` → `tag:graph-insights`. +## 1. Starting point + +Measured on the level-4 build of `main` (`dist/documentation-eccenca-com-26-2.pdf`). + +| Aspect | Today | +| :-- | :-- | +| Size | 1680 pages, A4, 118 MB | +| Type | body 10 pt Roboto Light (`size-base`, one step below the house style's measured 11 pt); tables 8.6 pt, admonitions and cards 9.1 pt, code 7.7 pt; 3 terminal tables shrink to fit, the widest (114 columns) to about 6.3 pt | +| Line length | about 104 characters per full body line (75th percentile, pages 40-80) | +| Parts | A Build pp. 3-734 (732) · B Explore 735-861 (127) · C Consume 862-900 (39) · D Distribution 901-910 (10) · E Deploy and Configure 911-1075 (165) · F Automate 1076-1192 (117) · G Develop 1193-1490 (298) · H Release Notes 1491-1677 (187) · I Tutorials 1678-1680 (3) | +| Largest blocks | A.3 Task and Operator Reference pp. 23-480 (458) · G Develop 298 · H Release Notes 187 | +| Geometry | symmetric margins: left/right 2.5 cm, top 3.9 cm, bottom 2.2 cm (`style.typ`, `set page`) | +| Header | logo left on every page; `Version 26.2` / `Generated ()` right on every page but page 1 | +| Footer | running title (`A Build › A.3 …`) left, `Page n \| total` right, orange, on every page but page 1 | +| Pagination | page 1 title page, page 2 parts contents; a part starts on whichever page follows | +| Spacing | paragraph spacing 2.24 em, block spacing 1.52 em (house style) | +| Links | 3193 internal jumps, 614 external links plus 70 into the published site; blue, underlined, external ones with `↗`; no footnotes | +| Images | 641, all ICC-based RGB; 565 carry a soft mask (transparency). Effective resolution in the print scope (§5), 586 images: 331 below 300 ppi, 144 below 200, 49 below 150 | +| Colour | orange, peach, slate and link blue; grey code ground; red for danger admonitions; the part diagrams mark their part with a red frame | +| Text | justified, hyphenation off | +| Tables | 273 header rows emitted by pandoc as `table.header`, which Typst repeats on every page | +| PDF | Typst 0.15.1 writes PDF 1.4-2.0, PDF/A and PDF/UA, **not PDF/X**; Ghostscript 10.08 is available (§7) | + +Verified in a Typst 0.15.1 probe: `set page(binding: left, margin: (inside: …, outside: …))`, +`pagebreak(to: "odd")`, `calc.odd(here().page())` in header and footer, `text(costs: (widow: …, +orphan: …))` and `footnote`. A blank page inserted by `pagebreak(to: "odd")` **still gets header and +footer**; suppressing them needs its own rule. + +## 2. Provider: BoD + +Decided (D2). Researched 2026-09-14; re-check the current specification before ordering. + +| Requirement | BoD | Source | +| :-- | :-- | :-- | +| Pages, A4 paperback | 1,200 on 80 g, 1,050 on 90 g white, 900 on 90 g cream, 120 g or 130 g | Buch drucken | +| Paper | white or cream 80 g is **for publishers only**; 90 g for everyone | Grafiken und Farbmanagement | +| Image resolution | at least **300 dpi** at printed size; line art without grey levels (bitmap) **1200 dpi** | Grafiken und Farbmanagement | +| Grey areas | at least **20 % black**, not lighter; grey steps in charts at least 20 % apart | Grafiken und Farbmanagement | +| Transparency | to be reduced before delivery; otherwise BoD reduces it, which "can change objects" | Grafiken und Farbmanagement | +| Colour | RGB recommended - BoD converts to CMYK for its presses; untagged RGB is read as sRGB, CMYK as ISO Coated v2 (FOGRA39); no colour separation | Grafiken und Farbmanagement | +| Bleed | 5 mm per side, only for content that runs off the page; no crop marks | Grafiken und Farbmanagement | +| Fonts and images | embedded | Hilfe | + +Sources: +[BoD Buch drucken](https://www.bod.de/buch-drucken.html), +[BoD Grafiken und Farbmanagement](https://www.bod.de/bodfiles/GLOBAL-Storage/documents/help-documents/bod-grafiken-und-farbmanagement.pdf), +[BoD Hilfe](https://www.bod.de/hilfe/hilfe-und-service.html). + +For comparison, not pursued: Amazon KDP allows 828 pages (black ink on white), Lulu 800. + +## 3. Requirements + +### R1 - Page furniture for verso and recto pages + +- Page numbers sit on the **outer edge**: left on even (verso) pages, right on odd (recto) pages. +- Running titles mirror too, following book convention (the larger unit on the left page): + - verso: `1234` at the outer left, `A Build` beside it + - recto: `A.3 Task and Operator Reference` beside `1235` at the outer right +- `| total` is dropped - it has no meaning on paper. +- No logo and no version stamp on text pages (D9); both stay on the title page and the imprint. +- Margins become `inside`/`outside` with `binding: left` on A4 (D7). The gutter must grow with the + spine; a starting proposal is inside 3.0 cm, outside 2.0 cm, which keeps today's 16 cm text width. +- No header and no footer on the title page, the imprint, blank pages and part covers. + +### R2 - The title page is page 1 of the book block + +- Page 1, recto. It keeps the house title block (eyebrow, title, version) and names the publisher, + **eccenca GmbH**. +- The site link and the copyright line move to the imprint. +- Counted as page 1, number not printed. + +### R3 - Imprint on page 2 + +Page 2, verso, no page furniture. Contents: + +- title and edition: `eccenca Corporate Memory - Documentation, Version 26.2`, print edition, + generated `` from commit `` (moves here from the running header) +- publisher: **eccenca GmbH, Hainstraße 8, 04109 Leipzig, Germany** +- authors: the list below +- licence: CC BY-SA 4.0, as `README.md` states, with the copyright line from `mkdocs.yml` +- online edition: `https://documentation.eccenca.com/26.2/`, named as the complete reference for what + the print edition shortens (§5) +- colophon: typeset with Typst from the Markdown sources; fonts Roboto, Roboto Mono, Noto Color + Emoji, DejaVu +- no ISBN (D3) + +#### Authors + +Rule (D4): GitHub accounts of the contributors to +[eccenca/documentation.eccenca.com](https://github.com/eccenca/documentation.eccenca.com), **most +commits first**, printed as their **GitHub IDs**. Not listed: anonymous contributions (commits whose +e-mail maps to no GitHub account), bot accounts and software agents (codex, claude). Ties are ordered by +ID, case-insensitive. + +Measured with `gh api repos/eccenca/documentation.eccenca.com/contributors` on 2026-09-14: + +| # | Commits | GitHub ID | +| --: | --: | :-- | +| 1 | 639 | rpietzsch | +| 2 | 532 | seebi | +| 3 | 66 | irangareddy | +| 4 | 62 | muddymudskipper | +| 5 | 61 | mgns | +| 6 | 53 | sobo | +| 7 | 29 | annamakor | +| 8 | 25 | msaipraneeth | +| 9 | 22 | saradaimi786 | +| 10 | 20 | louiswesterheide | +| 11 | 18 | robertisele | +| 12 | 12 | spl0tt | +| 13 | 11 | white-gecko | +| 14 | 10 | BorderCloud | +| 15 | 10 | haschek | +| 16 | 9 | MaximilianWenzel | +| 17 | 7 | tomatophantastico | +| 18 | 5 | adelahaye-ecc | +| 19 | 1 | dgrtner-ecc | +| 20 | 1 | looooph | +| 21 | 1 | peterfreytag | +| 22 | 1 | pkgut | + +Dropped: two anonymous entries (33 and 2 commits). + +- **No software agent is in the list today.** Claude's commits carry a `Co-Authored-By` trailer and + count for the human author; GitHub does not list co-authors as contributors. The exclusion rule + still belongs in the tooling. +- GitHub counts commits on the default branch only. + +### R4 - Links on paper + +In the print edition (D8): internal links print their text plus a page reference `(p. 34)`; external +links print their text with a footnote holding the URL; no colour, underline or arrow. Link annotations +are left out of the book block - Ghostscript refuses PDF/X output while a page carries one (§7). + +## 4. Decisions + +All made 2026-09-14. + +| # | Question | Decision | +| :-- | :-- | :-- | +| D1 | Scope | The print edition reduces sections instead of printing them in full, **configurable per section** as `full`, `list` or `omit`. Default: A.3 Task and Operator Reference → `list`, H Release Notes → `list`, everything else `full` (§5) | +| D2 | Provider | **BoD** (§2) | +| D3 | ISBN | none for now; may come later | +| D4 | Authors | GitHub IDs, most commits first, no anonymous entries, no bots or agents (§3) | +| D5 | Colour and paper | black and white interior on 80 g paper. BoD offers 80 g to publishers only - eccenca GmbH needs a publisher account | +| D6 | Editions | separate screen and print PDFs. `task pdf` stays the screen edition, unchanged; `task pdf:print` builds the book block, switched by a Typst input (`edition=print`) and a print configuration | +| D7 | Trim size | A4 | +| D8 | Links on paper | page references and URL footnotes, no link styling (R4) | +| D9 | Logo and version in the running header | title page and imprint only (R1) | +| D10 | Body type size | keep 10 pt and tighten the spacing (§5). A smaller body, 9 pt or even 8 pt, only if the page limit is still exceeded - not needed for the default configuration | +| D11 | Normalization | an **optional** Ghostscript pass after Typst: PDF/X-4, CMYK, all images at 300 dpi (§7) | -**Tag chip class** — `md-tag md-tag-icon md-tag--` where `` is -`extra.tags[]` from `mkdocs.yml`. **Zensical already emits the backing CSS** -(`.md-tag.md-tag--beginners{--md-tag-icon:url(...)}`) on every page, so no CSS work is -needed. For tags absent from `extra.tags`, production omits `md-tag-icon` and the -`md-tag--` modifier, emitting a bare ``. 14 of the 45 tags in -use are unmapped — see §5. +## 5. Page budget -**Item title** — front-matter `title:` if present, otherwise the first `# ` heading in the -body. Verified on three pages; none of the sampled pages set `title:`, and all three -listing titles match their H1 exactly. +### Type size and spacing -**Item link** — relative from the listing page to the target, so it survives mike's -versioned `/latest/`, `/26.2/` prefixes. Never absolute. +Measured by compiling variants of the current book (`dist/pdf/book.typ`) with modified copies of +`style.typ`: -**Ordering** — from `mkdocs.yml`: -- tags within an un-filtered listing: casefolded tag name - (`listings_tags_sort_by: tag_name_casefold`) — confirmed, `/tags/` runs - AdvancedTutorial, API, Application View, Automate, BeginnersTutorial… -- items within a tag: page title (`listings_sort_by: item_title`) -- markers with `include:` render in **marker order**, not sorted — `/tutorials/` shows - Beginners, Advanced, Expert, matching source order +| Variant | Pages | Characters per line | +| :-- | --: | --: | +| today: 10 pt, paragraph spacing 2.24 em, block spacing 1.52 em | 1680 | 104 | +| body 9 pt | 1512 (−10 %) | 115 | +| paragraph spacing 1.2 em, block spacing 1.0 em | 1571 (−6.5 %) | 105 | +| both | 1414 (−16 %) | 115 | -**Scope** — a bare `` lists every tag in use (45 on `/tags/`). An -`include: [X]` marker lists only tag X. +At 10 pt a full line already holds about 104 characters, well above the 45-75 recommended for +continuous reading; 9 pt makes it 115, and thinner light strokes on black-and-white 80 g paper make it +worse. The spacing saves two thirds as much without touching legibility. -**Self-inclusion** — not an issue: neither `tags.md` nor `tutorials/index.md` carries tag -front matter, so neither can appear in its own listing. Guard anyway. +A two-column A4 layout would bring lines to a comfortable length, but wide tables, code blocks and +screenshots would need to break out of the columns. Not proposed. -## 4. Corpus facts +A heavier weight than Light for the print body (Regular) may print more reliably in black and white; +measure its page cost before deciding. -- 531 of 581 pages carry `tags:` front matter -- 45 distinct tags; largest is `TransformOperator` (237 pages), then `WorkflowTask` (77), - `PythonPlugin` (61), `cmemc` (47) -- The `/tags/` listing is therefore large — production's is ~2100 words +### Section modes -## 5. Open questions — resolved - -1. **Unmapped tags.** 14 tags in use have no `extra.tags` icon: `Build`, `Dataset`, - `DistanceMeasure`, `EvaluateTemplate`, `Explainer`, `Files`, `Graph-Insights`, - `GraphInsights`, `Introduction`, `LinkRules`, `Load Balancer`, `Plugin`, - `TransformOperator`, `WorkflowTask`. Match production (render plain, no icon), or take - the opportunity to map them? **Recommend: match production**, keep this change - behaviour-neutral. -> decision: match production, no icon +`full` prints the section as today. `omit` leaves it out; the part contents point to the online +edition. `list` reduces it to a two-column table: -2. **`Graph-Insights` vs `GraphInsights`** are both in use and look like an accidental - split — 2 tags where 1 was meant. Out of scope for this task, but the listing will make - it visible on `/tags/`. Worth a separate content fix. -> decision: separate fix +| Section | `list` renders | Measured | +| :-- | :-- | :-- | +| A.3 Task and Operator Reference | the section's own page and its five overview pages (Aggregators, Custom Workflow Tasks, Datasets, Distance Measures, Transformers), which already are generated `Name \| Description` tables; the 176 individual operator pages are dropped, and the navigation's operator categories under Transformers print as one **Category** \| **Transformers** table instead of empty headings | 458 → about 17 pages | +| H Release Notes | one table: **Release** (e.g. `Corporate Memory 26.2.1`) and **Summary** (the release page's introductory paragraph; where a release has none, the components it lists, e.g. `DataIntegration v26.2.0, Explore v26.2.2, …`) | 22 releases, 187 → about 3 pages | -3. **`Load Balancer` icon.** The T7 work added `overrides/.icons/other/load-balancer.svg` - (committed, tracked) but the two `mkdocs.yml` lines that reference it are **not** in - `HEAD` — line 63 still reads `# "Load Balancer": simple-awselasticloadbalancing`. The - icon file is currently dead weight and the tag renders bare. Probably an accidental - partial revert. Restore those two lines, or drop the SVG? **Recommend: restore**, it - was verified working. -> decision: restore it, verify it works (again) - -4. **Failure mode.** If a marker is present but the script cannot render it, should - `task build` fail, or warn and leave the marker? **Recommend: fail**, consistent with - `localize_bundle_assets.py`, since a silently empty listing is the exact failure this - whole guard-rail effort exists to prevent. -> decision: fail - -5. **`/tags/` page size.** 45 sections listing 531 page references. Acceptable, or should - the bare marker exclude high-cardinality tags like `TransformOperator`? **Recommend: - include everything**, matching production. -> decision: match prod, include everything - -## 6. Known limitation: `task serve` - -`zensical serve` rebuilds into `site/` on every change, so it would overwrite anything a -post-build step writes. The live-reload preview therefore shows the raw -`` markers, not the listings - measured: 0 tag sections, 3 markers -left on `/tutorials/`. The same applies to the glightbox localisation. - -`task preview` covers the gap: it runs a full `task build` and serves the result on port -8001. Verified: 45 sections on `/tags/`, 3 on `/tutorials/`, 0 markers, 0 unpkg references. -This is inherent to post-build processing and goes away with the script. - -## 7. Removal - -The trigger is already wired. `check_zensical_output.py` tracks `tag-listings` as `PEND` -and prints a `NEW` banner the moment Zensical renders a listing itself. On that signal: - -1. delete `tools/render_tag_listings.py` -2. drop its line from `build` in `Taskfile.yml` -3. delete `overrides/partials/tags.html` if §9 was implemented - Zensical will populate - `tag.url` itself once it has a listing index, and its stock template already handles it -4. drop the `tag-listings-*` checks from `check_zensical_output.py` - -Sources need no changes, because they were never changed. +In `list` mode, links from the kept tables to dropped pages print as plain text, not as footnotes; the +section opens with a sentence naming the online reference URL. -## 8. Acceptance +The configuration lives in `tools/pdf/print.yml`, keyed by the section's path in `nav.yml` +(`build/reference/`, `release-notes/`), so a section can be switched without touching code. -- `/tags/` renders 45 tag sections; `/tutorials/` renders 3, in marker order -- zero `` comments remain in `site/**/*.html` -- `check_zensical_output.py` no longer reports unexpanded markers -- every generated link resolves to a real file in `site/` -- tag chips in listings render with icons for the 31 mapped tags -- `task build` still clean under `--strict`; page count unchanged at 584 -- output stable across two consecutive builds (no ordering nondeterminism) +### Estimate for the default configuration -## 9. Link page tag chips to their listing anchor +| Step | Pages | +| :-- | --: | +| today | 1680 | +| A.3 as `list` | ≈ 1239 | +| H Release Notes as `list` | ≈ 1055 | +| tighter spacing (−6.5 %) | ≈ 985 | +| imprint, recto starts and blank pages (at most 28) | ≈ 1015 | -**Status:** implemented 2026-08-23 as `overrides/partials/tags.html`. All three open -questions settled as recommended; the reachability assertion is `tag-chips-resolve` in -`check_zensical_output.py`. +About 1,015 pages: within BoD's 1,200 for A4 on 80 g. + +## 6. Further changes towards a print-ready book block -### Problem +### Pagination -Rendering the listings only solved half the feature. Material also turns each per-page tag -chip into a link to its section on `/tags/`, so a reader can jump from "this page is tagged -Docker" to "everything tagged Docker". Zensical emits the chips as inert ``s. +- Parts start on a recto page: cover recto, then contents recto, then the part's text recto + (`pagebreak(to: "odd")`). Costs at most three blank pages per part, 27 in total. +- The front contents start on page 3, recto. +- Blank pages stay empty - no header, no footer, no page number. +- The final page count is even. -Measured on the current build: **531 pages carry a tag nav, 703 chips in total, 0 of them -links.** Production, for the same page: +### Typography + +- Hyphenation on for the print edition (`lang: "en"`); justified text without it opens wide gaps. +- Widow and orphan costs, so no single line is left at the top or bottom of a page. +- Minimum type size: code 7.7 pt and the 6.3 pt terminal tables are acceptable on A4; set a floor in + `codeblock()` so a wider table cannot shrink below 6 pt. +- Hairlines: table rules are 0.3 pt, dividers 0.5 pt - above the usual 0.25 pt minimum. +- Tighter paragraph and block spacing (§5). + +### Black and white -```html -Configuration -``` +- **Grey areas at least 20 % black** (BoD). Today's grounds are far lighter: the code and note ground + `ec-wash` (`#F3F5F6`) and the peach bands print as a few percent grey. Either raise them to 20 % or + drop the fills in favour of rules. +- Orange footer text, the link blue and the admonition accents need values that stay distinguishable + in grey; the orange running title in particular turns into a pale grey. +- The part diagrams mark their part with a red frame; in grey the frame stays visible but no longer + stands out. Check the four diagrams. +- Screenshots print in grey; UI states that differ only by colour lose their meaning. -versus ours: +### Images -```html -Configuration -``` +- In the print scope 331 of 586 images are below 300 ppi at their printed size, 49 below 150. +- Stop scaling small screenshots up to the text width, which lowers their resolution further - the + cheapest way to bring images closer to 300 ppi. +- 565 images carry transparency, almost all of them PNG screenshots with an alpha channel over a white + page. BoD wants transparency reduced before delivery, and PDF/X-4 does not remove it (§7). Composite + the alpha channel onto white while normalizing images, before Typst embeds them - lossless for these + images, and the book block then carries no transparency at all. -Now that §1-§8 generate those anchors, every one of these links has a valid destination. -The anchors exist; nothing points at them. +### PDF output -### Approach: template override, *not* post-build injection +- The optional normalization pass (§7). +- A preflight report per build: page size, page count against 1,200 and parity, all fonts embedded + (`pdffonts`), images below 300 ppi (`pdfimages -list`), soft masks left, page-number position per page + (`pdftotext -bbox`), blank pages really blank. -`zensical/templates/partials/tags.html` is **byte-identical to Material 9.7.7's**. It -already contains the branch that produces a link: +### Content -```jinja -{% if tag.url %} - {{ tag.name }} -{% else %} - {{ tag.name }} -{% endif %} -``` +- Web-only phrasing: "click here", "Next chapter:" links, embedded videos printed as + `[iframe: ]`. Print them as a footnote, or as a QR code for the few videos. +- Tutorial step headings that number themselves print as `A.9.3 1 Install …` - a content cleanup. +- An index at the back from the front-matter tags (45 tags on 531 pages) would give a paper reader a + second way in. Optional. -The template is not the problem - `tag.url` is simply never populated, because Zensical has -no listing index to point at. So the fix is to compute the URL in an override rather than -depend on `tag.url`, exactly as `overrides/partials/tabs-item.html` does for tab icons. +### Delivery -This is strictly better than extending `render_tag_listings.py`: +- CI builds the book block as a second artifact next to the screen PDF. +- The cover (§9) needs the final page count and paper for its spine width, so it comes last. -| | template override | post-build injection | -| :-- | :-- | :-- | -| Works in `task serve` | **yes** | no (see §6) | -| Extra build step | none | another pass over 531 pages | -| Removal when #38 lands | delete one file | more code to unpick | +## 7. Optional normalization pass (Ghostscript) -### Verified feasible +An optional post-processing step on the print PDF: `task pdf:print` runs it when asked +(`--normalize` / `PDF_NORMALIZE=1`) and writes `…-print-x4.pdf` next to the unnormalized file. BoD +accepts either; the normalized file is what a provider preflight sees. -A spike produced output byte-identical to production's, then was reverted. MiniJinja -supports the three filters needed (`url`, `lower`, `replace`): +### What it does -```jinja -{% set anchor = "tags/" | url ~ "#tag:" ~ (tag.name | lower | replace(" ", "-")) %} -{{- tag.name -}} -``` +| Target | How | +| :-- | :-- | +| PDF/X-4 | `-dPDFX=4` with a prefix file derived from Ghostscript's `lib/PDFX_def.ps`, which writes `/GTS_PDFXVersion (PDF/X-4)`, `/Trapped /False` and the output intent | +| CMYK | `-sColorConversionStrategy=CMYK -sProcessColorModel=DeviceCMYK`; the output intent profile is ISO Coated v2 (FOGRA39), the space BoD assumes for CMYK | +| Images at 300 dpi | down: `-dDownsampleColorImages=true -dColorImageDownsampleType=/Bicubic -dColorImageResolution=300 -dColorImageDownsampleThreshold=1.0`, the same for grey images; lossless re-encoding with `/FlateEncode`. **Up: not possible in Ghostscript** (below) | +| No annotations | `-dPreserveAnnots=false` | -`| url` resolves `tags/` relative to the current page, so it survives mike's versioned -prefixes without special handling. +### Verified with Ghostscript 10.08.0 -### Open questions +Converted pages 3-40 of the current PDF on 2026-09-14: -1. **Slug duplication.** The template computes the slug in MiniJinja; `render_tag_listings. - tag_slug()` computes it in Python. They must agree or every link dangles. Accept the - duplication with a comment in both places, or have the renderer assert that each anchor - it emits is reachable? **Recommend: assert** - it is a cheap required check and the - failure mode is otherwise silent. +- the file carries `/GTS_PDFXVersion (PDF/X-4)` in the document info and `pdfxid:GTS_PDFXVersion='PDF/X-4'` + in XMP, one output intent, `/Trapped /False`, no annotations, all 154 fonts embedded +- all 7 images converted from ICC-based RGB to CMYK; the 610 ppi image came out at 300 ppi +- the 215 ppi image **stayed at 215 ppi: Ghostscript downsamples, it never upsamples** +- all 7 soft masks survived - PDF/X-4 keeps transparency +- the Build cover rendered from the CMYK file shows no visible colour shift against the RGB original at + screen resolution; the red frame, the orange and the peach band stay distinct +- 38 pages in 6 s: the whole book block takes a few minutes +- Ghostscript 10.08's documentation (`doc/src/VectorDevices.rst`) confirms `-dPDFX` values 1, 3 and 4 + (default 3) and PDF/X-4 support; the rendered manual page still says X-1 and X-3 only + +Traps found: + +- **A page with any annotation makes Ghostscript revert to normal PDF output**, with only a one-line + warning. The book has about 3,900 link annotations - drop them (`-dPreserveAnnots=false`) and fail the + task if the output lacks the PDF/X marker. +- `-dPDFX=4` alone writes **no** PDF/X marker and no output intent; the prefix file is required. +- `-sOutputICCProfile` with `-dPDFX` crashes the PDF interpreter (`/undefined in --runpdf--`) and leaves a + truncated file - supply the profile only through the prefix file, which reads it with + `--permit-file-read=`. +- Ghostscript ships a generic `default_cmyk.icc`, not FOGRA39. The ISO Coated v2 profile comes from the + ECI and has to be downloaded and vendored (with its licence) or fetched in CI. + +### Upsampling to 300 dpi + +Ghostscript cannot raise an image's resolution. Images below 300 ppi at their printed size have to be +resampled **before** Typst embeds them, in the build: + +- the printed width is known at build time for the common case: an image wider than the text column + prints at the text width (16 cm), a narrower one at its natural size +- the build writes a resampled copy (Lanczos) to `dist/pdf/images/` with enough pixels for 300 ppi at + that width, and points Typst at it; the originals under `site/` stay untouched +- the same step composites alpha onto white (§6), so resampling and transparency reduction happen once + +Upsampling adds no detail - BoD says so itself: an image does not get better by raising its resolution +in a graphics program. It gives the printer a 300 dpi image to rasterize instead of leaving the +interpolation to the press, and it makes the preflight report clean. The originals below 150 ppi still +need replacing (P14). + +### Black and white + +The book prints in black and white, but BoD converts RGB itself and the pass is specified as CMYK. A +greyscale variant (`-sColorConversionStrategy=Gray`, output intent from a grey profile) would show the +final tones on screen and give smaller files. Keep CMYK as specified; add greyscale as a preview mode of +the same task, for checking the palette (P11). -2. **Tags with no listing section.** A tag carried *only* by the listing page itself - produces no section (self-exclusion, §3), so its chip would link to a dead anchor. Does - not occur today. **Recommend: cover it with the check in Q1** rather than special-casing. +## 8. Acceptance -3. **`hide: tags` pages.** The existing template already honours it; the override must keep - that branch. No decision needed, just do not drop it. +- `task pdf` builds the screen PDF exactly as before: 1680 pages, unchanged look +- `task pdf:print` builds the book block with the configuration in `tools/pdf/print.yml` +- page 1 is the title page naming eccenca GmbH as publisher; page 2 is the imprint with the publisher's + address and the authors from `tools/pdf/authors.yml`, most commits first +- on every numbered page the page number sits at the outer edge: left on even pages, right on odd + pages - checked by the preflight report, not by eye +- title page, imprint, part covers and blank pages carry no header and no footer; text pages carry no + logo and no version stamp +- the front contents and every part cover start on an odd page +- A.3 and H Release Notes print as two-column lists in the default configuration; switching either to + `full` or `omit` in `print.yml` works without code changes +- internal links print page references, external links footnotes; the book block has no link annotations +- page count even and at most 1,200 +- grey areas at least 20 % black; all fonts embedded; no soft masks left +- every image at 300 ppi or more at its printed size, or listed and accepted +- with normalization: the file declares PDF/X-4, carries a FOGRA39 output intent, CMYK images only, no + image above 300 ppi +- `task check` and `task test:unit` pass + +## 9. Out of scope + +- the cover file: front, spine and back, spine width from page count and paper, barcode +- ISBN and retail distribution (D3, maybe later) +- EPUB or other e-book formats +- translations diff --git a/tests/test_build_pdf.py b/tests/test_build_pdf.py index 639731c53..b5229759f 100644 --- a/tests/test_build_pdf.py +++ b/tests/test_build_pdf.py @@ -24,6 +24,7 @@ def soup(markup: str) -> BeautifulSoup: def test_section_opens_with_its_index_page(): + # A page keeps its navigation title; an index page takes its section's. nav = [ {"Build": [ {"Build": "build/index.md"}, @@ -32,10 +33,10 @@ def test_section_opens_with_its_index_page(): ]}, ] assert nav_entries(nav) == [ - NavEntry(depth=0, md="build/index.md"), - NavEntry(depth=1, md="build/rules/index.md"), - NavEntry(depth=2, md="build/rules/linking.md"), - NavEntry(depth=1, md="build/spark.md"), + NavEntry(depth=0, md="build/index.md", title="Build"), + NavEntry(depth=1, md="build/rules/index.md", title="Rules"), + NavEntry(depth=2, md="build/rules/linking.md", title="Linking"), + NavEntry(depth=1, md="build/spark.md", title="Spark"), ] @@ -50,13 +51,13 @@ def test_section_without_index_page_gets_a_heading(): assert nav_entries(nav) == [ NavEntry(depth=0, title="Release Notes"), NavEntry(depth=1, title="2026"), - NavEntry(depth=2, md="release-notes/2026/corporate-memory-26-2/index.md"), + NavEntry(depth=2, md="release-notes/2026/corporate-memory-26-2/index.md", title="v26.2.1"), ] def test_page_listed_twice_appears_once_and_links_are_skipped(): nav = [{"A": ["a/index.md", {"Again": "a/index.md"}, {"Site": "https://example.org/"}]}] - assert nav_entries(nav) == [NavEntry(depth=0, md="a/index.md")] + assert nav_entries(nav) == [NavEntry(depth=0, md="a/index.md", title="A")] def test_merge_breaks_chapters_adds_section_headings_and_drops_web_chrome(tmp_path): diff --git a/tests/test_build_pdf_images.py b/tests/test_build_pdf_images.py new file mode 100644 index 000000000..07cd4cf38 --- /dev/null +++ b/tests/test_build_pdf_images.py @@ -0,0 +1,83 @@ +"""Test the print edition's image copies: on white, at 300 ppi at the printed width""" +from collections import Counter +from pathlib import Path + +from bs4 import BeautifulSoup +from PIL import Image + +from tools.build_pdf import print_image, resolve_images + + +def make(path: Path, size: tuple[int, int], dpi: tuple[int, int] | None = None) -> None: + image = Image.new("RGB", size, (0, 0, 255)) + image.save(path, **({"dpi": dpi} if dpi else {})) + + +def test_a_wide_screenshot_prints_at_the_text_width_with_300_ppi(tmp_path): + source = tmp_path / "wide.png" + make(source, (4000, 1000)) + with Image.open(print_image(source, None, tmp_path / "cache")) as copy: + # 16 cm are 453.54 pt, which at 300 ppi take 1890 pixels. + assert copy.size == (1890, 472) + assert round(copy.info["dpi"][0]) == 300 + + +def test_a_small_screenshot_keeps_its_printed_size_and_gains_pixels(tmp_path): + source = tmp_path / "small.png" + make(source, (300, 100), dpi=(144, 144)) + with Image.open(print_image(source, None, tmp_path / "cache")) as copy: + # At 144 dpi it prints 150 pt wide, which at 300 ppi take 625 pixels. + assert copy.size == (625, 208) + assert round(copy.info["dpi"][0]) == 300 + + +def test_a_percentage_width_is_that_share_of_the_text_width(tmp_path): + source = tmp_path / "half.png" + make(source, (2000, 1000)) + with Image.open(print_image(source, "50%", tmp_path / "cache")) as copy: + assert copy.width == 945 + + +def test_transparency_is_flattened_onto_white(tmp_path): + source = tmp_path / "alpha.png" + image = Image.new("RGBA", (200, 100), (0, 0, 0, 0)) + image.putpixel((0, 0), (255, 0, 0, 255)) + image.save(source) + with Image.open(print_image(source, None, tmp_path / "cache")) as copy: + assert copy.mode == "RGB" + assert copy.getpixel((copy.width - 1, copy.height - 1)) == (255, 255, 255) + + +def test_a_gif_becomes_a_png_of_its_first_frame(tmp_path): + source = tmp_path / "animation.gif" + frames = [Image.new("P", (100, 50), index) for index in (1, 2)] + frames[0].save(source, save_all=True, append_images=frames[1:]) + copy = print_image(source, None, tmp_path / "cache") + assert copy.suffix == ".png" + with Image.open(copy) as image: + assert image.mode == "RGB" + + +def test_the_copy_is_reused_while_the_image_and_its_width_stay_the_same(tmp_path): + source = tmp_path / "shot.png" + make(source, (900, 300)) + first = print_image(source, None, tmp_path / "cache") + written = first.stat().st_mtime_ns + assert print_image(source, None, tmp_path / "cache") == first + assert first.stat().st_mtime_ns == written + assert print_image(source, "50%", tmp_path / "cache") != first + + +def test_print_images_point_at_their_copies_and_vector_images_stay(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + site = Path("site") + (site / "build").mkdir(parents=True) + make(site / "build" / "shot.png", (900, 300)) + (site / "build" / "chart.svg").write_text("") + doc = BeautifulSoup('ShotChart', "html.parser") + stats: Counter = Counter() + resolve_images(doc, site, stats, [], Path("dist/pdf/print/images")) + shot, chart = doc.find_all("img") + assert shot["src"].startswith("/dist/pdf/print/images/") and shot["src"].endswith(".png") + assert chart["src"] == "/site/build/chart.svg" + assert stats["images normalized for print"] == 1 diff --git a/tests/test_build_pdf_print.py b/tests/test_build_pdf_print.py new file mode 100644 index 000000000..018b1031c --- /dev/null +++ b/tests/test_build_pdf_print.py @@ -0,0 +1,248 @@ +"""Test the print edition's section modes: full, list and omit""" +import click +import pytest + +from tools.build_pdf import ( + Generated, + NavEntry, + SectionRule, + apply_section_rules, + load_section_rules, + md_to_built_html, + merge_pages, +) + +REFERENCE = [ + NavEntry(0, "build/index.md"), + NavEntry(1, "build/reference/index.md"), + NavEntry(2, "build/reference/aggregator/index.md"), + NavEntry(3, "build/reference/aggregator/average.md"), + NavEntry(2, "build/reference/transformer/index.md"), + NavEntry(3, "build/reference/transformer/Numeric/index.md"), + NavEntry(4, "build/reference/transformer/Numeric/add.md"), + NavEntry(1, "build/spark.md"), +] + +# The shape of Release Notes in nav.yml: no page stands for the section or its years. +RELEASES = [ + NavEntry(0, title="Release Notes"), + NavEntry(1, title="2026"), + NavEntry(2, "release-notes/2026/cm-26-2/index.md"), + NavEntry(2, "release-notes/2026/cm-26-1/index.md"), + NavEntry(1, title="2025"), + NavEntry(2, "release-notes/2025/cm-25-3/index.md"), + NavEntry(0, "tutorials/index.md"), +] + +# The shape of Transformers in nav.yml: the overview page lists every operator, +# and the categories are titles without a page. +CATEGORIES = [ + NavEntry(1, "build/reference/index.md", "Task and Operator Reference"), + NavEntry(2, "build/reference/transformer/index.md", "Transformers"), + NavEntry(3, title="Combine"), + NavEntry(4, "build/reference/transformer/Combine/concat.md", "Concatenate"), + NavEntry(4, "build/reference/transformer/Combine/zip.md", "Zip"), + NavEntry(3, title="Date"), + NavEntry(4, "build/reference/transformer/Date/duration.md"), + NavEntry(1, "build/spark.md", "Spark"), +] +CATEGORY_TABLE = Generated( + "groups", "build/reference/", "list", columns=("Category", "Transformers"), + groups=[("Combine", [CATEGORIES[3], CATEGORIES[4]]), ("Date", [CATEGORIES[6]])], +) + + +def test_rules_read_a_mode_or_a_mapping_with_columns(tmp_path): + path = tmp_path / "print.yml" + path.write_text( + "sections:\n" + " build/reference/: list\n" + " release-notes:\n" + " mode: list\n" + " columns: [Release, Summary]\n" + " tutorials/: full\n" + ) + assert load_section_rules(path) == [ + SectionRule("build/reference/", "list"), + SectionRule("release-notes/", "list", ("Release", "Summary")), + SectionRule("tutorials/", "full"), + ] + + +def test_an_unknown_mode_fails(tmp_path): + path = tmp_path / "print.yml" + path.write_text("sections:\n build/: shorten\n") + with pytest.raises(click.ClickException, match="shorten"): + load_section_rules(path) + + +def test_list_keeps_the_overview_pages_and_drops_what_they_list(): + entries, dropped = apply_section_rules(REFERENCE, [SectionRule("build/reference/", "list")]) + assert entries == [ + NavEntry(0, "build/index.md"), + NavEntry(1, "build/reference/index.md"), + NavEntry(1, generated=Generated("note", "build/reference/", "list", ["build/reference/index.md"])), + NavEntry(2, "build/reference/aggregator/index.md"), + NavEntry(2, "build/reference/transformer/index.md"), + NavEntry(1, "build/spark.md"), + ] + assert dropped == { + "build/reference/aggregator/average.md", + "build/reference/transformer/Numeric/index.md", + "build/reference/transformer/Numeric/add.md", + } + + +def test_list_without_overview_pages_makes_a_table_under_each_heading(): + rule = SectionRule("release-notes/", "list", ("Release", "Summary")) + entries, dropped = apply_section_rules(RELEASES, [rule]) + first = "release-notes/2026/cm-26-2/index.md" + assert entries == [ + NavEntry(0, title="Release Notes"), + NavEntry(1, generated=Generated("note", "release-notes/", "list", [first], name="Release Notes")), + NavEntry(1, title="2026"), + NavEntry(2, generated=Generated( + "table", "release-notes/", "list", [first, "release-notes/2026/cm-26-1/index.md"], ("Release", "Summary") + )), + NavEntry(1, title="2025"), + NavEntry(2, generated=Generated( + "table", "release-notes/", "list", ["release-notes/2025/cm-25-3/index.md"], ("Release", "Summary") + )), + NavEntry(0, "tutorials/index.md"), + ] + assert len(dropped) == 3 + + +def test_list_turns_the_titles_an_overview_empties_into_one_table(): + entries, dropped = apply_section_rules(CATEGORIES, [SectionRule("build/reference/", "list")]) + assert entries == [ + CATEGORIES[0], + NavEntry(1, generated=Generated("note", "build/reference/", "list", ["build/reference/index.md"])), + CATEGORIES[1], + NavEntry(3, generated=CATEGORY_TABLE), + CATEGORIES[7], + ] + assert dropped == {CATEGORIES[3].md, CATEGORIES[4].md, CATEGORIES[6].md} + + +def test_omit_drops_the_pages_and_leaves_a_heading_with_a_note(): + entries, dropped = apply_section_rules(REFERENCE, [SectionRule("build/reference/", "omit")]) + assert entries == [ + NavEntry(0, "build/index.md"), + NavEntry(1, generated=Generated("note", "build/reference/", "omit", ["build/reference/index.md"])), + NavEntry(1, "build/spark.md"), + ] + assert len(dropped) == 6 + + +def test_omitting_a_whole_part_without_a_page_leaves_nothing_of_it(): + entries, dropped = apply_section_rules(RELEASES, [SectionRule("release-notes/", "omit")]) + assert entries == [NavEntry(0, "tutorials/index.md")] + assert len(dropped) == 3 + + +def test_full_changes_nothing(): + assert apply_section_rules(RELEASES, [SectionRule("release-notes/", "full")]) == (RELEASES, set()) + + +def test_a_section_without_pages_in_the_navigation_fails(): + with pytest.raises(click.ClickException, match="build/nothing/"): + apply_section_rules(REFERENCE, [SectionRule("build/nothing/", "list")]) + + +def write_page(site, md, body): + path = site / md_to_built_html(md) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(f'
{body}
') + + +def test_merge_renders_notes_and_tables_and_unlinks_what_the_section_drops(tmp_path): + site = tmp_path / "site" + write_page(site, "build/reference/index.md", '

Task and Operator Reference

All operators.

') + write_page( + site, "build/reference/aggregator/index.md", + '

Aggregators

Average
' + '

See Spark.

', + ) + write_page( + site, "release-notes/2026/cm-26-2/index.md", + '

Corporate Memory 26.2.1

The second major release.

' + '

DataIntegration v26.2.0

', + ) + write_page( + site, "release-notes/2026/cm-26-1/index.md", + '

Corporate Memory 26.1.3

no paragraph
' + '

DataIntegration v26.1.0

Explore v26.1.3

', + ) + first, second = "release-notes/2026/cm-26-2/index.md", "release-notes/2026/cm-26-1/index.md" + entries = [ + NavEntry(1, "build/reference/index.md"), + NavEntry(1, generated=Generated("note", "build/reference/", "list", ["build/reference/index.md"])), + NavEntry(2, "build/reference/aggregator/index.md"), + NavEntry(0, title="Release Notes"), + NavEntry(1, generated=Generated("note", "release-notes/", "list", [first], name="Release Notes")), + NavEntry(1, title="2026"), + NavEntry(2, generated=Generated("table", "release-notes/", "list", [first, second], ("Release", "Summary"))), + ] + rules = [SectionRule("build/reference/", "list"), SectionRule("release-notes/", "list", ("Release", "Summary"))] + dropped = {"build/reference/aggregator/average.md", first, second} + doc, missing = merge_pages(entries, site, "https://example.org/26.2/", rules, dropped) + assert missing == [] + html = str(doc.body) + text = " ".join(doc.body.get_text(" ").split()) + + # The overview's link to a dropped operator page prints as its text; a link + # to a page outside the print edition still leads to the published site. + assert "Average" in html + assert 'href="https://example.org/26.2/build/spark/"' in html + + # A note names the section and where it is complete: the section's own URL + # when it has a page, else the first page it lists. + assert ( + "This print edition lists the Task and Operator Reference in short. The complete section is part of " + "the online edition: https://example.org/26.2/build/reference/" + ) in text + assert ( + "This print edition lists the Release Notes in short. The complete section is part of the online " + "edition: https://example.org/26.2/release-notes/2026/cm-26-2/" + ) in text + + # The table: each page's title and first paragraph, without links; a page + # without a paragraph is summarized by its headings. + table = doc.find("table", class_="print-list") + rows = [[cell.get_text(" ", strip=True) for cell in row.find_all(["th", "td"])] for row in table.find_all("tr")] + assert rows == [ + ["Release", "Summary"], + ["Corporate Memory 26.2.1", "The second major release."], + ["Corporate Memory 26.1.3", "DataIntegration v26.1.0, Explore v26.1.3"], + ] + assert table.find("a") is None + + +def test_merge_renders_the_emptied_titles_as_one_table_of_their_pages(tmp_path): + site = tmp_path / "site" + write_page(site, "build/reference/transformer/Date/duration.md", '

Duration

') + doc, _ = merge_pages([NavEntry(3, generated=CATEGORY_TABLE)], site, "https://example.org/26.2/") + table = doc.find("table", class_="print-list") + rows = [[cell.get_text(" ", strip=True) for cell in row.find_all(["th", "td"])] for row in table.find_all("tr")] + # A page without a navigation title is named by its own title. + assert rows == [ + ["Category", "Transformers"], + ["Combine", "Concatenate, Zip"], + ["Date", "Duration"], + ] + assert doc.body.find(["h1", "h2", "h3", "h4", "h5", "h6"]) is None + # Without an alignment pandoc would centre the cells. + assert {cell.get("style") for cell in table.find_all(["th", "td"])} == {"text-align: left;"} + + +def test_merge_renders_an_omitted_section_as_its_title_and_a_note(tmp_path): + site = tmp_path / "site" + write_page(site, "build/reference/index.md", '

Task and Operator Reference

') + entries = [NavEntry(1, generated=Generated("note", "build/reference/", "omit", ["build/reference/index.md"]))] + doc, _ = merge_pages(entries, site, "https://example.org/26.2/", [SectionRule("build/reference/", "omit")], set()) + assert doc.body.find("h2").get_text() == "Task and Operator Reference" + assert ( + "The Task and Operator Reference is not part of this print edition. It is part of the online edition: " + "https://example.org/26.2/build/reference/" + ) in " ".join(doc.body.get_text(" ").split()) diff --git a/tests/test_pdf_authors.py b/tests/test_pdf_authors.py new file mode 100644 index 000000000..ec5046811 --- /dev/null +++ b/tests/test_pdf_authors.py @@ -0,0 +1,45 @@ +"""Test the author list of the print edition's imprint""" +import yaml + +from tools.pdf_authors import authors_yaml, next_page, select_authors + + +def test_most_commits_first_and_ties_by_id_case_insensitive(): + contributors = [ + {"login": "haschek", "contributions": 10, "type": "User"}, + {"login": "seebi", "contributions": 532, "type": "User"}, + {"login": "BorderCloud", "contributions": 10, "type": "User"}, + {"login": "rpietzsch", "contributions": 639, "type": "User"}, + ] + assert [a["id"] for a in select_authors(contributors)] == ["rpietzsch", "seebi", "BorderCloud", "haschek"] + + +def test_bots_agents_and_anonymous_contributions_are_not_authors(): + contributors = [ + {"login": "dependabot[bot]", "contributions": 40, "type": "Bot"}, + {"login": "claude-agent", "contributions": 30, "type": "User"}, + {"login": "Codex", "contributions": 20, "type": "User"}, + {"name": "Someone", "email": "someone@example.org", "contributions": 33, "type": "Anonymous"}, + {"login": "sobo", "contributions": 53, "type": "User"}, + ] + assert select_authors(contributors) == [{"id": "sobo", "commits": 53}] + + +def test_pagination_follows_the_next_link_only(): + header = ( + '; rel="next", ' + '; rel="last"' + ) + assert next_page(header) == "https://api.github.com/repositories/1/contributors?per_page=100&page=2" + assert next_page("") is None + + +def test_the_file_names_its_source_and_keeps_the_order(): + text = authors_yaml("eccenca/docs", [{"id": "b", "commits": 2}, {"id": "a", "commits": 1}]) + assert text.startswith("---\n# Generated by `dec-tool pdf-authors`") + # yamllint's default rules want list items indented under their key. + assert "authors:\n - id: b\n commits: 2\n" in text + assert yaml.safe_load(text) == { + "repository": "eccenca/docs", + "authors": [{"id": "b", "commits": 2}, {"id": "a", "commits": 1}], + } diff --git a/tests/test_update_di_reference.py b/tests/test_update_di_reference.py index 6328c512a..0ee2cdcfa 100644 --- a/tests/test_update_di_reference.py +++ b/tests/test_update_di_reference.py @@ -29,13 +29,13 @@ def test_get_plugin_descriptions(): assert titles == sorted(titles) -def _make_plugin(plugin_id, plugin_type="transformer", main_category="Extract", related=None): +def _make_plugin(plugin_id, plugin_type="transformer", main_category="Extract", related=None, description="test plugin"): """Build a minimal valid PluginDescription for a test, filling in only what varies.""" return PluginDescription( pluginId=plugin_id, title=plugin_id, categories=[main_category], - description="test plugin", + description=description, properties={}, actions={}, required=[], @@ -45,6 +45,31 @@ def _make_plugin(plugin_id, plugin_type="transformer", main_category="Extract", ) +@pytest.mark.parametrize( + "description, expected", + [ + # the Excel map transformer: a fence without a language + ( + "sheets of the form:\n```\nmapFrom,mapTo\n,\n```\nand more.", + "sheets of the form: `mapFrom,mapTo ,` and more.", + ), + # a fence with a language + ("Run:\n```sparql\nSELECT *\nWHERE { ?s ?p ?o }\n```", "Run: `SELECT * WHERE { ?s ?p ?o }`"), + # a fence already on one line + ("Use ```a,b``` here.", "Use `a,b` here."), + # a backtick inside needs a longer delimiter + ("Quote:\n```\nx = `y`\n```", "Quote: `` x = `y` ``"), + # inline code stays as it is + ("Insert `=\"\"` in a cell.", "Insert `=\"\"` in a cell."), + ], +) +def test_description_turns_a_fenced_block_into_a_code_span(description, expected): + # A fence squeezed onto one line is no fence in Markdown, but rumdl --fix + # still takes it for one and inserts a blank line before it - which ends the + # operator table the description is a row of. + assert _make_plugin("excelMap", description=description).description == expected + + @pytest.mark.parametrize( "current_path, target_path, expected", [ diff --git a/tools/__init__.py b/tools/__init__.py index 3e42a61e4..b38c1f2a2 100644 --- a/tools/__init__.py +++ b/tools/__init__.py @@ -6,6 +6,7 @@ from tools.build_pdf import build_pdf from tools.check_zensical_output import check_zensical_output from tools.localize_bundle_assets import localize_bundle_assets +from tools.pdf_authors import pdf_authors from tools.publish import publish from tools.update_di_reference import update_di_reference from tools.update_icons import update_icons @@ -19,6 +20,7 @@ def cli(): cli.add_command(build_pdf) cli.add_command(check_zensical_output) cli.add_command(localize_bundle_assets) +cli.add_command(pdf_authors) cli.add_command(publish) cli.add_command(update_icons) cli.add_command(update_di_reference) diff --git a/tools/build_pdf.py b/tools/build_pdf.py index 197858b3a..2d2ba7619 100644 --- a/tools/build_pdf.py +++ b/tools/build_pdf.py @@ -8,12 +8,20 @@ converted to Typst by pandoc with `tools/pdf/filter.lua`, and typeset by Typst with the eccenca house style in `tools/pdf/style.typ`. -Invoked by `task pdf`. Every option falls back to an environment variable, so -BUILD_VERSION, PDF_OUT, PANDOC and TYPST keep working as tunables. +`--edition print` builds the book block of a printed book instead of the screen +PDF: the same pipeline, shortened by the section modes in `tools/pdf/print.yml`, +with images copied at 300 ppi without transparency, and typeset by the print +branches of the style (tasks/spec.md). + +Invoked by `task pdf` and `task pdf:print`. Every option falls back to an +environment variable, so BUILD_VERSION, PDF_EDITION, PDF_OUT, PANDOC and TYPST +keep working as tunables. """ from __future__ import annotations import base64 +import hashlib +import io import re import shutil import subprocess @@ -21,13 +29,14 @@ import time import urllib.parse from collections import Counter -from dataclasses import dataclass +from dataclasses import dataclass, field from datetime import date from pathlib import Path import click import yaml from bs4 import BeautifulSoup, NavigableString +from PIL import Image DEFAULT_OUT_STEM = "dist/documentation-eccenca-com" SITE_DIR = Path("site") @@ -35,6 +44,8 @@ MKDOCS_YML = Path("mkdocs.yml") # Style, pandoc template, filter, logo and vendored fonts. PDF_ASSETS = Path("tools/pdf") +# Publisher and section modes of the print edition. +PRINT_YML = PDF_ASSETS / "print.yml" # The merged HTML and the Typst source pandoc writes, kept for debugging. WORK_DIR = Path("dist/pdf") CONTENT_SELECTOR = "article.md-content__inner" @@ -56,15 +67,61 @@ TESTED_TYPST = "0.15" BOOK_TITLE = "eccenca Corporate Memory" BOOK_CONTEXT = "Documentation" +# The screen PDF, and the book block of the printed book (tasks/spec.md). +EDITIONS = ("screen", "print") +# How the print edition prints a navigation section (tools/pdf/print.yml). +SECTION_MODES = ("full", "list", "omit") +DEFAULT_COLUMNS = ("Page", "Summary") +# A `groups` table: the navigation titles under an overview page - the operator +# categories of the reference - and their pages, under the overview's title. +GROUP_COLUMNS = ("Category", "Pages") +# The print edition's text column is 16 cm wide, and BoD asks for images at +# 300 dpi at their printed size (tasks/spec.md, §2). +TEXT_WIDTH_PT = 16 / 2.54 * 72 +PRINT_PPI = 300 + + +@dataclass +class Generated: + """Content the print edition sets in place of a section's pages. + + A `note` names the online edition for a section in `list` or `omit` mode; a + `table` lists pages that no overview page of the section lists; `groups` + lists the navigation titles whose pages an overview page lists, each with + the names of its pages. + """ + + kind: str + section: str + mode: str + pages: list[str] = field(default_factory=list) + columns: tuple[str, str] = DEFAULT_COLUMNS + # The section's name when no page of it carries one. + name: str | None = None + # The rows of a `groups` table: a navigation title and the page entries beneath it. + groups: list[tuple[str, list[NavEntry]]] = field(default_factory=list) @dataclass class NavEntry: - """One stop along the navigation: a page, or the title of a section that has none.""" + """One stop along the navigation: a page, the title of a section that has none, or generated content. + + A page carries the title the navigation gives it, when it gives one. + """ depth: int md: str | None = None title: str | None = None + generated: Generated | None = None + + +@dataclass +class SectionRule: + """How the print edition prints the pages under one docs/ directory.""" + + prefix: str + mode: str + columns: tuple[str, str] = DEFAULT_COLUMNS def load_site_config() -> dict: @@ -106,14 +163,16 @@ def nav_entries(nav: list) -> list[NavEntry]: depth. A section without one - Release Notes and its years - gets its title as a heading of its own; otherwise its pages would read as part of whichever chapter came before. Everything else in a section sits one level below it. + A page keeps the title the navigation gives it; an index page takes its + section's. """ entries: list[NavEntry] = [] seen: set[str] = set() - def add_page(md: str, depth: int) -> None: + def add_page(md: str, depth: int, title: str | None = None) -> None: if md.endswith(".md") and md not in seen: seen.add(md) - entries.append(NavEntry(depth=depth, md=md)) + entries.append(NavEntry(depth=depth, md=md, title=title)) def add_section(title: str, children: list, depth: int) -> None: index = section_index(children[0]) if children else None @@ -121,7 +180,7 @@ def add_section(title: str, children: list, depth: int) -> None: entries.append(NavEntry(depth=depth, title=str(title))) add_items(children, depth + 1) else: - add_page(index, depth) + add_page(index, depth, str(title)) add_items(children[1:], depth + 1) def add_items(items: list, depth: int) -> None: @@ -131,7 +190,7 @@ def add_items(items: list, depth: int) -> None: elif isinstance(item, dict): for title, value in item.items(): if isinstance(value, str): - add_page(value, depth) + add_page(value, depth, str(title)) elif isinstance(value, list): add_section(title, value, depth) @@ -145,6 +204,193 @@ def load_nav_entries(nav_yml: Path = NAV_YML) -> list[NavEntry]: return nav_entries(data.get("nav") or []) +# -- print edition: section modes ------------------------------------------------ +# The print edition fits the page limit of a printed book by shortening sections +# that are reference material online (tools/pdf/print.yml). `list` keeps a +# section's overview pages and drops the pages they list; pages no overview +# lists become a table of title and first paragraph, and navigation titles left +# without pages become one table of titles and page names. `omit` drops a +# section and leaves a note naming the online edition. + + +def load_section_rules(path: Path = PRINT_YML) -> list[SectionRule]: + """The section modes of the print edition. + + A mode is given either as the value of a section, or as `mode` in a mapping + that may also name the two `columns` of the section's tables. + """ + data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + rules = [] + for prefix, value in (data.get("sections") or {}).items(): + spec = value if isinstance(value, dict) else {"mode": value} + mode = spec.get("mode", "full") + if mode not in SECTION_MODES: + raise click.ClickException( + f"{path}: section {prefix} has mode {mode!r}, not one of {', '.join(SECTION_MODES)}" + ) + columns = tuple(spec.get("columns") or DEFAULT_COLUMNS) + if len(columns) != 2: + raise click.ClickException(f"{path}: section {prefix} needs two columns, not {len(columns)}") + rules.append(SectionRule(str(prefix).rstrip("/") + "/", mode, columns)) + return rules + + +def is_heading_entry(entry: NavEntry) -> bool: + """A section title the navigation gives without a page.""" + return entry.md is None and entry.generated is None + + +def section_name(prefix: str) -> str: + """A readable name for a section that has neither a page nor a navigation title.""" + return prefix.rstrip("/").rsplit("/", 1)[-1].replace("-", " ").replace("_", " ").title() + + +def list_section(entries: list[NavEntry], rule: SectionRule) -> tuple[list[NavEntry], set[str]]: + """Keep a section's own page and its subsections' overview pages; table what no overview lists. + + The navigation titles left without pages are tabled too (`group_listed_titles`). + """ + root = rule.prefix + "index.md" + overview = re.compile(re.escape(rule.prefix) + r"[^/]+/index\.md$") + kept = {e.md for e in entries if e.md and (e.md == root or overview.match(e.md))} + # An overview page lists the pages in and below its own directory. + listed_by = [md[: -len("index.md")] for md in kept] + result: list[NavEntry] = [] + dropped: set[str] = set() + listed: set[str] = set() + table: Generated | None = None + for entry in entries: + if not (entry.md and entry.md.startswith(rule.prefix)): + result.append(entry) + table = None + continue + if entry.md in kept: + result.append(entry) + table = None + if entry.md == root: + result.append(NavEntry(entry.depth, generated=Generated("note", rule.prefix, "list", [root]))) + continue + dropped.add(entry.md) + if any(entry.md.startswith(directory) for directory in listed_by): + listed.add(entry.md) + continue + if table is None: + table = Generated("table", rule.prefix, "list", [], rule.columns) + result.append(NavEntry(entry.depth, generated=table)) + table.pages.append(entry.md) + result = group_listed_titles(entries, result, listed, rule) + + if root not in kept: + first = next((i for i, e in enumerate(result) if e.generated and e.generated.section == rule.prefix), None) + if first is not None: + # The note opens the section: before the titles that lead into its + # first table, after the part title they belong to. + start = first + while start > 0 and is_heading_entry(result[start - 1]) and result[start - 1].depth > 0: + start -= 1 + before = result[start - 1] if start > 0 else None + name = before.title if before is not None and is_heading_entry(before) else section_name(rule.prefix) + pages = result[first].generated.pages[:1] + note = Generated("note", rule.prefix, "list", pages, name=name) + result.insert(start, NavEntry(result[start].depth, generated=note)) + return result, dropped + + +def group_listed_titles( + entries: list[NavEntry], result: list[NavEntry], listed: set[str], rule: SectionRule +) -> list[NavEntry]: + """Replace each run of navigation titles whose pages an overview lists by one table. + + Without their pages such titles would print as empty headings - the + operator categories after the Transformers overview do. The table has a row + per title naming the pages beneath it; the overview before the run names the + column of pages. Titles are told apart by identity, as two can read the same. + """ + beneath: dict[int, list[NavEntry]] = {} + for i, entry in enumerate(entries): + if is_heading_entry(entry): + pages: list[NavEntry] = [] + for below in entries[i + 1:]: + if below.depth <= entry.depth: + break + if below.md: + pages.append(below) + beneath[id(entry)] = pages + + def emptied(index: int) -> bool: + entry = result[index] + pages = beneath.get(id(entry), []) if is_heading_entry(entry) else [] + following = result[index + 1] if index + 1 < len(result) else None + return ( + bool(pages) + and all(page.md in listed for page in pages) + and (following is None or following.depth <= entry.depth) + ) + + grouped: list[NavEntry] = [] + i = 0 + while i < len(result): + if not emptied(i): + grouped.append(result[i]) + i += 1 + continue + depth = result[i].depth + rows = [] + while i < len(result) and result[i].depth == depth and emptied(i): + rows.append((result[i].title, beneath[id(result[i])])) + i += 1 + overview = next((e for e in reversed(grouped) if e.md and e.depth < depth), None) + pages_column = overview.title if overview is not None and overview.title else GROUP_COLUMNS[1] + table = Generated("groups", rule.prefix, "list", columns=(GROUP_COLUMNS[0], pages_column), groups=rows) + grouped.append(NavEntry(depth, generated=table)) + return grouped + + +def drop_empty_headings(entries: list[NavEntry]) -> list[NavEntry]: + """Remove section titles that no longer have anything beneath them.""" + result: list[NavEntry] = [] + for entry in reversed(entries): + following = result[-1] if result else None + if is_heading_entry(entry) and (following is None or following.depth <= entry.depth): + continue + result.append(entry) + return list(reversed(result)) + + +def omit_section(entries: list[NavEntry], rule: SectionRule) -> tuple[list[NavEntry], set[str]]: + """Drop a section. A section with a page of its own leaves its title and a note in its place.""" + root = rule.prefix + "index.md" + result: list[NavEntry] = [] + dropped: set[str] = set() + for entry in entries: + if entry.md and entry.md.startswith(rule.prefix): + dropped.add(entry.md) + if entry.md == root: + result.append(NavEntry(entry.depth, generated=Generated("note", rule.prefix, "omit", [root]))) + continue + result.append(entry) + return drop_empty_headings(result), dropped + + +def apply_section_rules(entries: list[NavEntry], rules: list[SectionRule]) -> tuple[list[NavEntry], set[str]]: + """Shorten the navigation by the section modes. Returns the entries and the pages dropped from them.""" + dropped: set[str] = set() + for rule in rules: + if not any(e.md and e.md.startswith(rule.prefix) for e in entries): + raise click.ClickException(f"{PRINT_YML}: no page in {NAV_YML} lies under {rule.prefix}") + if rule.mode == "list": + entries, gone = list_section(entries, rule) + elif rule.mode == "omit": + entries, gone = omit_section(entries, rule) + else: + continue + dropped |= gone + return entries, dropped + + +# -- merging --------------------------------------------------------------------- + + def md_to_built_html(src_md: str) -> str: """Map a docs/ markdown path to its built HTML (use_directory_urls=True).""" if src_md == "index.md": @@ -200,13 +446,17 @@ def namespace_ids(section, section_id: str) -> None: a["href"] = f"#{section_id}-{a['href'][1:]}" -def resolve_links(section, base_url: str, valid_ids: set, public_base: str) -> None: +def resolve_links( + section, base_url: str, valid_ids: set, public_base: str, unlink_ids: set | frozenset = frozenset() +) -> None: """Turn page-relative URLs into something valid inside the merged document. - Links to other documented pages become in-PDF anchor jumps. Everything - else relative - downloadable resources, screenshots opened at full size, - pages outside the navigation - becomes an absolute link into the published - site, which is the only address a reader of the PDF can follow. + Links to other documented pages become in-PDF anchor jumps. Links to a page + in `unlink_ids` - one the print edition drops from the section being merged + - print as their text. Everything else relative - downloadable resources, + screenshots opened at full size, pages outside the navigation - becomes an + absolute link into the published site, which is the only address a reader of + the PDF can follow. Image sources become root-absolute site paths; they are embedded, not linked. """ @@ -227,6 +477,8 @@ def resolve_links(section, base_url: str, valid_ids: set, public_base: str) -> N if parsed.fragment else f"#{section_id}" ) + elif section_id and section_id in unlink_ids: + a.unwrap() else: a["href"] = urllib.parse.urljoin(public_base, absolute.lstrip("/")) @@ -272,21 +524,159 @@ def part_cover(doc: BeautifulSoup, section) -> None: node.unwrap() +def page_article(site_dir: Path, md: str): + """A page's built article with its web chrome dropped, or None when the page is not built.""" + built = site_dir / md_to_built_html(md) + if not built.exists(): + return None + article = BeautifulSoup(built.read_text(encoding="utf-8"), "html.parser").select_one(CONTENT_SELECTOR) + if article is not None: + drop_noise(article) + return article + + +def plain_heading(heading) -> str: + """A heading's text without the icons some titles carry.""" + for icon in heading.select(".twemoji"): + icon.decompose() + return " ".join(heading.get_text(" ", strip=True).split()) + + +def page_title(site_dir: Path, md: str) -> str: + article = page_article(site_dir, md) + heading = article.find("h1") if article is not None else None + return plain_heading(heading) if heading is not None else md + + +def list_cell(doc: BeautifulSoup, name: str = "td"): + """A cell of a print-list table, set left: pandoc centres a cell that has no alignment.""" + return doc.new_tag(name, attrs={"style": "text-align: left;"}) + + +def list_table(doc: BeautifulSoup, columns: tuple[str, str]): + """An empty print-list table under its two column labels, and the body to fill.""" + table = doc.new_tag("table", attrs={"class": "print-list"}) + head = doc.new_tag("thead") + row = doc.new_tag("tr") + for label in columns: + cell = list_cell(doc, "th") + cell.string = label + row.append(cell) + head.append(row) + body = doc.new_tag("tbody") + table.append(head) + table.append(body) + return table, body + + +def summary_table(doc: BeautifulSoup, generated: Generated, site_dir: Path): + """A two-column table of pages: the title, and the paragraph that opens the page. + + A page that opens with something else is summarized by its second-level + headings, which on a release page name the components released. + """ + table, body = list_table(doc, generated.columns) + for md in generated.pages: + article = page_article(site_dir, md) + heading = article.find("h1") if article is not None else None + row = doc.new_tag("tr") + title = list_cell(doc) + title.string = plain_heading(heading) if heading is not None else md + summary = list_cell(doc) + opening = heading.find_next_sibling() if heading is not None else None + if opening is not None and opening.name == "p": + for link in opening.find_all("a"): + link.unwrap() + for image in opening.find_all("img"): + image.decompose() + for child in list(opening.contents): + summary.append(child.extract()) + elif article is not None: + summary.string = ", ".join(plain_heading(h) for h in article.find_all("h2")) + row.append(title) + row.append(summary) + body.append(row) + return table + + +def groups_table(doc: BeautifulSoup, generated: Generated, site_dir: Path): + """A two-column table of navigation titles: the title, and the names of the pages beneath it. + + A page is named by its navigation title, or by its own title when the + navigation gives it none. + """ + table, body = list_table(doc, generated.columns) + for title, pages in generated.groups: + row = doc.new_tag("tr") + for text in (title, ", ".join(page.title or page_title(site_dir, page.md) for page in pages)): + cell = list_cell(doc) + cell.string = text + row.append(cell) + body.append(row) + return table + + +def section_url(generated: Generated, site_dir: Path, public_base: str) -> str: + """Where a shortened section is complete: its own page online, else the first page it lists.""" + if (site_dir / generated.section / "index.html").is_file(): + path = generated.section + else: + path = md_to_url_path(generated.pages[0]).lstrip("/") + return urllib.parse.urljoin(public_base, path) + + +def render_generated(doc: BeautifulSoup, entry: NavEntry, site_dir: Path, public_base: str) -> list: + """The elements that stand in for a shortened section's pages.""" + generated = entry.generated + if generated.kind == "table": + return [summary_table(doc, generated, site_dir)] + if generated.kind == "groups": + return [groups_table(doc, generated, site_dir)] + name = generated.name or page_title(site_dir, generated.pages[0]) + url = section_url(generated, site_dir, public_base) + elements = [] + if generated.mode == "omit": + heading = doc.new_tag(f"h{min(6, entry.depth + 1)}") + heading.string = name + elements.append(heading) + text = f"The {name} is not part of this print edition. It is part of the online edition: {url}" + else: + text = f"This print edition lists the {name} in short. The complete section is part of the online edition: {url}" + note = doc.new_tag("div", attrs={"class": "admonition info"}) + paragraph = doc.new_tag("p") + paragraph.string = text + note.append(paragraph) + elements.append(note) + return elements + + def merge_pages( - entries: list[NavEntry], site_dir: Path, public_base: str + entries: list[NavEntry], + site_dir: Path, + public_base: str, + rules: list[SectionRule] | None = None, + dropped: set[str] | None = None, ) -> tuple[BeautifulSoup, list[str]]: """Merge the built articles along the navigation into one HTML document. Every top-level entry is a part. It is preceded by a chapter break, so a part starts on a new page, and its page opens with the part's cover (see `part_cover`); a part without a page gets its title and the contents marker. - Returns the document and the navigation pages that had no built HTML. + Generated entries of the print edition's section modes are rendered where + they stand, and a shortened section's links to the pages it drops print as + text. Returns the document and the navigation pages that had no built HTML. """ doc = BeautifulSoup('', "html.parser") valid_ids = {url_to_section_id(md_to_url_path(e.md)) for e in entries if e.md} + dropped_ids = {url_to_section_id(md_to_url_path(md)) for md in dropped or ()} + shortened = [rule.prefix for rule in rules or () if rule.mode != "full"] missing: list[str] = [] for entry in entries: + if entry.generated is not None: + for element in render_generated(doc, entry, site_dir, public_base): + doc.body.append(element) + continue if entry.depth == 0: doc.body.append(doc.new_tag("div", attrs={"class": "chapter-break"})) if entry.md is None: @@ -297,24 +687,20 @@ def merge_pages( heading.insert_after(doc.new_tag("div", attrs={"class": "part-contents"})) continue - built = site_dir / md_to_built_html(entry.md) - article = None - if built.exists(): - page = BeautifulSoup(built.read_text(encoding="utf-8"), "html.parser") - article = page.select_one(CONTENT_SELECTOR) + # page_article drops the web chrome before the ids are namespaced: + # selectors such as #__comments would no longer match afterwards. + article = page_article(site_dir, entry.md) if article is None: missing.append(entry.md) continue - # Before the ids are namespaced: selectors such as #__comments would no - # longer match afterwards. - drop_noise(article) section_id = url_to_section_id(md_to_url_path(entry.md)) section = doc.new_tag("section", attrs={"class": "print-page", "id": section_id}) for child in list(article.children): section.append(child.extract()) namespace_ids(section, section_id) - resolve_links(section, md_to_url_path(entry.md), valid_ids, public_base) + unlink_ids = dropped_ids if any(entry.md.startswith(prefix) for prefix in shortened) else frozenset() + resolve_links(section, md_to_url_path(entry.md), valid_ids, public_base, unlink_ids) demote_headings(section, entry.depth) if entry.depth == 0: part_cover(doc, section) @@ -458,14 +844,54 @@ def typst_path(path: Path) -> str: return "/" + path.resolve().relative_to(Path.cwd().resolve()).as_posix() -def resolve_images(doc: BeautifulSoup, site_dir: Path, stats: Counter, warnings: list[str]) -> None: - """Point images at the built files Typst embeds, and replace what it cannot embed. +def printed_width_pt(image: Image.Image, width: str | None) -> float: + """The width Typst prints an image at in the print edition's text column. + + A percentage is that share of the column. Without one, Typst sizes an image + by the pixel density it declares - 72 dpi when it declares none - and never + wider than the column. + """ + if width and width.strip().endswith("%"): + return TEXT_WIDTH_PT * float(width.strip()[:-1]) / 100 + dpi = image.info.get("dpi") + density = float(dpi[0]) if dpi and dpi[0] else 72.0 + return min(image.width * 72 / density, TEXT_WIDTH_PT) + + +def print_image(source: Path, width: str | None, cache_dir: Path) -> Path: + """A copy of an image for the print edition: flattened onto white, at 300 ppi at its printed width. + + BoD asks for 300 dpi and no transparency. The copy declares 300 dpi, so Typst + prints it at the width the original printed at, resampled up or down with + Lanczos. Copies are cached by the original's content and the pixel width. + """ + data = source.read_bytes() + with Image.open(io.BytesIO(data)) as image: + pixels = max(1, round(printed_width_pt(image, width) / 72 * PRINT_PPI)) + target = cache_dir / f"{hashlib.sha256(data).hexdigest()[:16]}-{pixels}.png" + if target.is_file(): + return target + image.seek(0) + frame = image.convert("RGBA") + flat = Image.new("RGB", frame.size, "white") + flat.paste(frame, mask=frame.getchannel("A")) + height = max(1, round(frame.height * pixels / frame.width)) + cache_dir.mkdir(parents=True, exist_ok=True) + flat.resize((pixels, height), Image.Resampling.LANCZOS).save(target, "PNG", dpi=(PRINT_PPI, PRINT_PPI)) + return target + + +def resolve_images( + doc: BeautifulSoup, site_dir: Path, stats: Counter, warnings: list[str], print_images: Path | None = None +) -> None: + """Point images at the files Typst embeds, and replace what it cannot embed. Emoji that the site loads as Twemoji images become the emoji character, which the vendored emoji font draws. Other remote images - status badges - become their alt text: the build does not fetch from the network. An SVG that keeps its text in foreignObject elements, as Mermaid does by default, - renders without that text in Typst, so it is reported. + renders without that text in Typst, so it is reported. With `print_images`, + raster images point at their print copies in that directory (`print_image`). """ for lightbox in doc.select("a.glightbox"): lightbox.unwrap() @@ -492,7 +918,11 @@ def resolve_images(doc: BeautifulSoup, site_dir: Path, stats: Counter, warnings: continue if path.suffix.lower() == ".svg" and " None: section.unwrap() -def normalize(doc: BeautifulSoup, site_dir: Path) -> tuple[Counter, list[str]]: +def normalize(doc: BeautifulSoup, site_dir: Path, print_images: Path | None = None) -> tuple[Counter, list[str]]: """Rewrite the merged document into elements pandoc's HTML reader understands. Returns counts of what was rewritten and warnings about content the PDF @@ -552,7 +982,7 @@ def normalize(doc: BeautifulSoup, site_dir: Path) -> tuple[Counter, list[str]]: pair_tabs(doc, stats) mark_cards(doc, stats) inline_icons(doc, stats) - resolve_images(doc, site_dir, stats, warnings) + resolve_images(doc, site_dir, stats, warnings, print_images) embedded_media_to_links(doc, stats) settle_internal_links(doc, stats) return stats, warnings @@ -613,6 +1043,24 @@ def run(command: list[str], what: str) -> None: print(f"{what} took {time.monotonic() - started:.0f}s") +def unpadded_pages(compile_command: list[str], typ_path: Path) -> int: + """How many pages the print edition has before it is padded to an even count. + + The style marks the book's last page with ``. Typst cannot add the + blank page itself: a page break that depends on the page count never lets the + layout converge. So the build asks first and passes `pad=true` when needed. + """ + started = time.monotonic() + command = [compile_command[0], "eval", "query().first().value", "--in", str(typ_path)] + command += compile_command[2:] + result = subprocess.run(command, capture_output=True, text=True, check=False) + if result.returncode != 0: + raise click.ClickException(f"typst eval failed: {result.stderr.strip()[:500]}") + pages = int(result.stdout.strip()) + print(f"typst eval took {time.monotonic() - started:.0f}s: {pages} pages before padding") + return pages + + @click.command() @click.option( "--build-version", @@ -621,12 +1069,20 @@ def run(command: list[str], what: str) -> None: help="Which version to stamp on the title page and fold into the file name?", show_default=True, ) +@click.option( + "--edition", "edition_name", + type=click.Choice(EDITIONS), + default="screen", + envvar="PDF_EDITION", + help="Which edition: the screen PDF, or the book block for print on demand?", + show_default=True, +) @click.option( "--output-file", "-o", type=click.Path(exists=False, dir_okay=False, file_okay=True), default=None, envvar="PDF_OUT", - help=f"Where to write the PDF? [default: {DEFAULT_OUT_STEM}-.pdf]", + help=f"Where to write the PDF? [default: {DEFAULT_OUT_STEM}-.pdf, -print.pdf for print]", ) @click.option( "--pandoc", "pandoc_binary", @@ -643,11 +1099,16 @@ def run(command: list[str], what: str) -> None: show_default=True, ) def build_pdf( - build_version: str, output_file: str | None, pandoc_binary: str, typst_binary: str + build_version: str, edition_name: str, output_file: str | None, pandoc_binary: str, typst_binary: str ) -> None: """Build a single PDF of the whole site with pandoc and Typst.""" version = build_version.strip() or "dev" - out = Path(output_file or f"{DEFAULT_OUT_STEM}-{version.replace('.', '-')}.pdf") + print_edition = edition_name == "print" + suffix = "-print" if print_edition else "" + out = Path(output_file or f"{DEFAULT_OUT_STEM}-{version.replace('.', '-')}{suffix}.pdf") + # The print edition merges and normalizes differently, so its intermediate + # files must not overwrite the screen edition's. + work_dir = WORK_DIR / "print" if print_edition else WORK_DIR if not (SITE_DIR / "index.html").exists(): raise click.ClickException(f"{SITE_DIR} is not built - run `task build` first.") @@ -665,11 +1126,17 @@ def build_pdf( entries = load_nav_entries() if not entries: raise click.ClickException(f"no pages found in {NAV_YML}") + rules = load_section_rules() if print_edition else [] + entries, dropped = apply_section_rules(entries, rules) + for rule in rules: + if rule.mode != "full": + count = sum(1 for md in dropped if md.startswith(rule.prefix)) + print(f"Print edition: {rule.prefix} as {rule.mode}, {count} pages dropped") config = load_site_config() public_base = public_base_url(config, version) - doc, missing = merge_pages(entries, SITE_DIR, public_base) + doc, missing = merge_pages(entries, SITE_DIR, public_base, rules, dropped) pages = sum(1 for e in entries if e.md) - len(missing) - headings = sum(1 for e in entries if e.md is None) + headings = sum(1 for e in entries if is_heading_entry(e)) if missing: print( f"WARNING: {len(missing)} page(s) in {NAV_YML} had no built HTML: " @@ -679,14 +1146,14 @@ def build_pdf( ) print(f"Merged {pages} pages and {headings} section headings along {NAV_YML}") - stats, warnings = normalize(doc, SITE_DIR) + stats, warnings = normalize(doc, SITE_DIR, work_dir / "images" if print_edition else None) for warning in warnings: print(f"WARNING: {warning}", file=sys.stderr) print("Normalized " + ", ".join(f"{count} {what}" for what, count in sorted(stats.items()))) - WORK_DIR.mkdir(parents=True, exist_ok=True) - html_path = WORK_DIR / "book.html" - typ_path = WORK_DIR / "book.typ" + work_dir.mkdir(parents=True, exist_ok=True) + html_path = work_dir / "book.html" + typ_path = work_dir / "book.typ" html_path.write_text(str(doc), encoding="utf-8") run( @@ -704,8 +1171,16 @@ def build_pdf( typst, "compile", "--root", ".", "--ignore-system-fonts", "--font-path", str(PDF_ASSETS / "fonts"), ] - for key, value in edition(config, version, public_base, date.today(), source_commit()).items(): + inputs = edition(config, version, public_base, date.today(), source_commit()) + # Only the print edition passes the switch, so the screen edition's Typst + # call is the one it always was. + if print_edition: + inputs["edition"] = "print" + for key, value in inputs.items(): command += ["--input", f"{key}={value}"] + if print_edition and unpadded_pages(command, typ_path) % 2: + # A printed book has an even page count; the style adds a blank last page. + command += ["--input", "pad=true"] run(command + [str(typ_path), str(out)], "typst") print(f"PDF written to {out} ({out.stat().st_size // 1024} KB)") diff --git a/tools/pdf/authors.yml b/tools/pdf/authors.yml new file mode 100644 index 000000000..7ff2db061 --- /dev/null +++ b/tools/pdf/authors.yml @@ -0,0 +1,49 @@ +--- +# Generated by `dec-tool pdf-authors` from the GitHub contributors of +# eccenca/documentation.eccenca.com. Do not edit - run `task pdf:authors`. +repository: eccenca/documentation.eccenca.com +authors: + - id: rpietzsch + commits: 639 + - id: seebi + commits: 532 + - id: irangareddy + commits: 66 + - id: muddymudskipper + commits: 62 + - id: mgns + commits: 61 + - id: sobo + commits: 53 + - id: annamakor + commits: 29 + - id: msaipraneeth + commits: 25 + - id: saradaimi786 + commits: 22 + - id: louiswesterheide + commits: 20 + - id: robertisele + commits: 18 + - id: spl0tt + commits: 12 + - id: white-gecko + commits: 11 + - id: BorderCloud + commits: 10 + - id: haschek + commits: 10 + - id: MaximilianWenzel + commits: 9 + - id: tomatophantastico + commits: 7 + - id: adelahaye-ecc + commits: 5 + - id: dgrtner-ecc + commits: 1 + - id: looooph + commits: 1 + - id: peterfreytag + commits: 1 + - id: pkgut + commits: 1 diff --git a/tools/pdf/filter.lua b/tools/pdf/filter.lua index 6d4b143c0..7d390af8d 100644 --- a/tools/pdf/filter.lua +++ b/tools/pdf/filter.lua @@ -37,7 +37,8 @@ function Div(el) -- Consumed by the enclosing admonition below. return nil elseif c:includes('chapter-break') then - return raw('#pagebreak(weak: true)') + -- A new page on screen, a right-hand page in print; style.typ decides. + return raw('#chapter-break()') elseif c:includes('part-contents') then return raw('#part-contents()') elseif c:includes('admonition') then diff --git a/tools/pdf/print.yml b/tools/pdf/print.yml new file mode 100644 index 000000000..3b6c17c5a --- /dev/null +++ b/tools/pdf/print.yml @@ -0,0 +1,30 @@ +--- +# The print edition of the PDF: `task pdf:print`, see tasks/spec.md. +# tools/build_pdf.py reads `sections`; tools/pdf/style.typ reads the rest. + +# Printed on the title page and in the imprint. +publisher: + name: eccenca GmbH + address: + - Hainstraße 8 + - 04109 Leipzig + - Germany + +# The weight of the body text: light, as on screen, or regular, which prints +# darker in black and white. +body-weight: light + +# How a navigation section is printed, keyed by the docs/ directory its pages +# live in. A section that is not listed prints in full. +# full - as on the site +# list - its overview: the section page and the overview pages of its +# subsections; pages no overview lists become a two-column table of +# title and first paragraph, under `columns` when given; navigation +# titles whose pages an overview lists become one table of the +# titles and their page names +# omit - left out; a note names the online edition +sections: + build/reference/: list + release-notes/: + mode: list + columns: [Release, Summary] diff --git a/tools/pdf/style.typ b/tools/pdf/style.typ index d10b3ea8c..03e07897c 100644 --- a/tools/pdf/style.typ +++ b/tools/pdf/style.typ @@ -12,14 +12,26 @@ // so the PDF does not depend on what is installed where it is built. // ============================================================================= +// -- edition -------------------------------------------------------------------- +// `dec-tool build-pdf --edition print` passes `--input edition=print` for the +// book block of a printed book (tasks/spec.md). Everything that differs between +// the editions branches on this flag; without the input the screen PDF is built +// exactly as before. +#let print-edition = sys.inputs.at("edition", default: "screen") == "print" + // -- palette (house style) ---------------------------------------------------- -#let ec-orange = rgb("#F29100") -#let ec-peach = rgb("#FCF0E0") -#let ec-slate = rgb("#2E3B45") -#let ec-grey = rgb("#6B757C") +// DEVIATION (print): the book prints in black and white on BoD's presses, which +// screen a grey below 20 % black unevenly or not at all. So text and accents +// print black, the orange turns a dark grey that still reads as a rule, the peach +// bands take the lightest grey BoD accepts, and the grounds lighter than that - +// code and note admonitions - are left white (see the rules that use ec-wash). +#let ec-orange = if print-edition { luma(30%) } else { rgb("#F29100") } +#let ec-peach = if print-edition { luma(80%) } else { rgb("#FCF0E0") } +#let ec-slate = if print-edition { luma(0%) } else { rgb("#2E3B45") } +#let ec-grey = if print-edition { luma(40%) } else { rgb("#6B757C") } #let ec-link = rgb("#0563C1") // DEVIATION: the policy palette has no alarm colour; danger admonitions need one. -#let ec-red = rgb("#B3261E") +#let ec-red = if print-edition { luma(0%) } else { rgb("#B3261E") } // DEVIATION: neutral ground for code and for note and info admonitions. #let ec-wash = rgb("#F3F5F6") @@ -78,7 +90,9 @@ #let admonition(kind: "note", title: none, body) = { let accent = if kind in alarm-kinds { ec-red } else if kind in caution-kinds { ec-orange } else { ec-slate } - let ground = if kind in alarm-kinds { ec-red.lighten(92%) } else if kind in caution-kinds { ec-peach } else { ec-wash } + // DEVIATION (print): no ground - the light grounds print below BoD's 20 % + // black - so the bar on the left carries the kind alone. + let ground = if print-edition { none } else if kind in alarm-kinds { ec-red.lighten(92%) } else if kind in caution-kinds { ec-peach } else { ec-wash } block( width: 100%, fill: ground, @@ -118,6 +132,8 @@ // the build passes the widest line in characters, and the text shrinks until // it fits. 0.61 em is the advance of Roboto Mono (0.600) and of the DejaVu Sans // Mono box glyphs (0.602), with a little room; 18pt is the block's inset. +// DEVIATION (print): the text never shrinks below 6 pt, the smallest size that +// stays legible on paper; a wider table then runs past the block's edge. #let codeblock(title: "", columns: none, body) = block(width: 100%, above: 1.2em, below: 1.2em, { if title != "" { block( @@ -136,7 +152,8 @@ } else { layout(region => { let fitting = (region.width - 18pt) / (columns * 0.61) - show raw.where(block: true): set text(size: calc.min(size-code, fitting)) + let size = calc.min(size-code, fitting) + show raw.where(block: true): set text(size: if print-edition { calc.max(6pt, size) } else { size }) body }) } @@ -170,15 +187,63 @@ it.body } +// -- print edition: right-hand starts and page furniture -------------------------- +// A bound book starts a part, its contents and its text on a right-hand (odd) +// page. The break leaves a blank left-hand page when it has to, and Typst gives +// that page the same header and footer as any other. So the break brackets +// itself with two markers: a page strictly between them was inserted blank. +#let recto-break() = { + [#metadata("recto-before") ] + pagebreak(weak: true, to: "odd") + [#metadata("recto-after") ] +} + +// Pages that carry neither header nor footer in the print edition: the title +// page, the imprint, the part covers and blank pages. +#let bare-page(page) = { + if page <= 2 { return true } + if query(heading.where(level: 1)).any(h => h.location().page() == page) { return true } + let pairs = query().zip(query()) + if pairs.any(((before, after)) => before.location().page() < page and page < after.location().page()) { + return true + } + query().any(marker => marker.location().page() == page) +} + +// DEVIATION (print): page numbers sit on the outer edge, as in a bound book. A +// left-hand page names the part beside its number, a right-hand page the page +// the reader is in. The page total means nothing on paper and is left out. +#let print-footer(page-no) = { + set par(justify: false, leading: 0.56em) + set text(size: size-meta, fill: ec-orange) + let started(h) = h.location().page() <= page-no + let parts = query(heading.where(level: 1)).filter(started) + let folio = text(weight: "medium", counter(page).display()) + let running = if parts.len() == 0 { none } else if calc.even(page-no) { parts.last() } else { + let pages = query(heading.where(level: 2).after(parts.last().location())).filter(started) + if pages.len() > 0 { pages.last() } else { parts.last() } + } + let title = if running == none { [] } else { numbered-title(running) } + if calc.even(page-no) { + grid(columns: (auto, 1fr), column-gutter: 1.2em, folio, title) + } else { + grid(columns: (1fr, auto), column-gutter: 1.2em, align(right, title), folio) + } +} + // The label above a contents. #let contents-title = text(size: size-h2, weight: "bold", fill: ec-slate)[Content] +// The break before a part, its contents and its text: a new page on screen, a +// right-hand page in print. tools/pdf/filter.lua places it before every part. +#let chapter-break() = if print-edition { recto-break() } else { pagebreak(weak: true) } + // DEVIATION: a policy has one contents. A part of this book has a cover page - // its title and, if the part has one, the diagram of where it sits - and its // own contents on the pages after: the part's pages and two levels of their // sections, up to the next part. The build places the call after the cover. #let part-contents() = { - pagebreak(weak: true) + chapter-break() contents-title v(0.4em) context { @@ -195,7 +260,38 @@ .after(here()).before(next-part), ) } - pagebreak(weak: true) + chapter-break() +} + +// DEVIATION (print): the back of the title page is the imprint - edition, +// publisher, authors, licence and the online edition - and it carries the +// edition stamp the screen PDF prints in its header. tools/pdf/print.yml holds +// the publisher and the section modes, tools/pdf/authors.yml the authors. +#let imprint(title: "", context-line: "", subtitle: "", generated: "", site-url: "", copyright: "") = { + let config = yaml("print.yml") + let mode(section) = if type(section) == dictionary { section.at("mode", default: "full") } else { section } + let shortened = config.at("sections", default: (:)).values().any(section => mode(section) != "full") + let authors = yaml("authors.yml").authors.map(author => author.id) + set text(size: size-small, fill: ec-slate) + set par(justify: false, leading: 0.6em, spacing: 1.2em) + let entry(label, body) = block(below: 1.3em, [#text(weight: "bold", label) \ #body]) + v(1fr) + entry(title)[#context-line, #subtitle \ Print edition, generated #generated] + entry("Publisher")[#config.publisher.name \ #config.publisher.address.join(linebreak())] + entry("Authors")[The contributors to the documentation by their GitHub IDs, most commits first: #authors.join(", ").] + // The URL is a string: written as markup, Typst would turn it into a link. + entry("Licence")[ + This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License, + #"https://creativecommons.org/licenses/by-sa/4.0/". #copyright + ] + entry("Online edition")[ + #site-url + #if shortened [\ The online edition is the complete reference; this print edition shortens some of its sections.] + ] + entry("Typesetting")[ + Typeset with Typst from the Markdown sources of the documentation, in Roboto, Roboto Mono, Noto Color + Emoji and DejaVu. + ] } // ----------------------------------------------------------------------------- @@ -224,28 +320,41 @@ // -- page furniture (house style geometry) ------------------------------------- set page( paper: "a4", - margin: (top: 3.9cm, bottom: 2.2cm, left: 2.5cm, right: 2.5cm), + // DEVIATION (print): a bound book mirrors its margins - wider inside for the + // binding, narrower outside - and keeps the 16 cm text width. Its header is + // empty, with no logo and no version on text pages, so the top margin + // shrinks to what a page without it needs. + binding: if print-edition { left } else { auto }, + margin: if print-edition { + (top: 2.5cm, bottom: 2.2cm, inside: 3.0cm, outside: 2.0cm) + } else { + (top: 3.9cm, bottom: 2.2cm, left: 2.5cm, right: 2.5cm) + }, header-ascent: 1.14cm, footer-descent: 0.51cm, // The title page carries version and date in its title block, so its // header is the logo alone. - header: context grid( - columns: (1fr, auto), - align(left + top, pad(left: -28pt, image("logo.svg", width: 4cm))), - if here().page() == 1 { [] } else { - align(right + bottom, text(size: size-meta, fill: ec-grey, weight: "regular", { - set par(leading: 0.56em) - [Version #version] - linebreak() - [Generated #generated] - })) - }, - ), + header: if print-edition { none } else { + context grid( + columns: (1fr, auto), + align(left + top, pad(left: -28pt, image("logo.svg", width: 4cm))), + if here().page() == 1 { [] } else { + align(right + bottom, text(size: size-meta, fill: ec-grey, weight: "regular", { + set par(leading: 0.56em) + [Version #version] + linebreak() + [Generated #generated] + })) + }, + ) + }, // DEVIATION: the policy footer repeats the document title. In a book the // chapter and the page the reader is in are the more useful running title. footer: context { let page-no = here().page() - if page-no > 1 { + if print-edition { + if not bare-page(page-no) { print-footer(page-no) } + } else if page-no > 1 { // A long page title wraps; spread to the full width, it would gape. set par(justify: false, leading: 0.56em) let started(h) = h.location().page() <= page-no @@ -267,18 +376,26 @@ ) // -- base typography (house style) ----------------------------------------------- + // DEVIATION (print): tools/pdf/print.yml may set a heavier body weight, which + // prints darker in black and white. + let body-weight = if print-edition { yaml("print.yml").at("body-weight", default: "light") } else { "light" } set text( font: sans, - weight: "light", + weight: body-weight, size: size-body, fill: ec-slate, lang: "en", - hyphenate: false, + // DEVIATION (print): justified text without hyphenation opens gaps that + // a printed page shows more plainly than a screen. Titles stay whole. + hyphenate: print-edition, // A glyph that none of the vendored fonts has falls back to the fonts // built into Typst instead of printing as an empty box. fallback: true, ) - set par(justify: true, leading: 0.76em, spacing: 2.24em) + show heading: set text(hyphenate: false) + // DEVIATION (print): the house spacing suits a two-page policy; over a + // thousand printed pages it costs a tenth of the book (tasks/spec.md, §5). + set par(justify: true, leading: 0.76em, spacing: if print-edition { 1.2em } else { 2.24em }) set list( indent: 1.84em, body-indent: 1.5em, @@ -295,24 +412,53 @@ }, ) set enum(indent: 1.84em, body-indent: 1.64em, spacing: 0.85em) - set block(spacing: 1.52em) - show link: it => { + set block(spacing: if print-edition { 1.0em } else { 1.52em }) + // DEVIATION (print): paper cannot follow a link. A link within the book prints + // the page it leads to, a link out of the book its address in a footnote, and + // neither leaves a link in the PDF (tasks/spec.md, R4). + show link: it => if not print-edition { text(fill: ec-link, underline(it)) if type(it.dest) == str and it.dest.starts-with(regex("https?://")) { external-mark } + } else if type(it.dest) == label { + // Always printed, also for a target on the same page: a reference that + // appears only across pages moves lines, which moves the target back, and + // the layout never settles. + it.body + context [~(p.~#numbering("1", ..counter(page).at(locate(it.dest))))] + } else if type(it.dest) == str { + let address = it.dest.trim("mailto:", at: start) + it.body + // An address that prints as its own text needs no footnote repeating it. + if it.body.at("text", default: none) != address { footnote(address) } + } else { + it.body + } + // Footnotes carry addresses: set small, and breakable after / . _ -. + show footnote.entry: set text(size: size-table) + show footnote.entry: it => { + show regex("[/._-]"): mark => mark + sym.zws + it } + // DEVIATION (print): contents entries without link annotations. + show outline.entry: it => if print-edition { block(it.indented(it.prefix(), it.inner())) } else { it } // -- code --------------------------------------------------------------------- show raw: set text(font: mono, weight: "regular") // No horizontal extent: the table rule below splits a code span into several // text runs at its break opportunities, and a run whose background reaches // past its own edge paints over the last glyph of the run before it. - show raw.where(block: false): it => highlight(fill: ec-wash, extent: 0pt, text(size: 0.84em, it)) + // DEVIATION (print): no ground behind code, which would print below BoD's 20 % + // black. The monospace face marks a code span, a thin frame a code block. + show raw.where(block: false): it => if print-edition { text(size: 0.84em, it) } else { + highlight(fill: ec-wash, extent: 0pt, text(size: 0.84em, it)) + } // The size is a show-set rule of its own, not part of the block below, so // that codeblock() can shrink a terminal table from an inner scope. show raw.where(block: true): set text(size: size-code) show raw.where(block: true): it => block( width: 100%, - fill: ec-wash, + fill: if print-edition { none } else { ec-wash }, + stroke: if print-edition { 0.5pt + ec-grey } else { none }, inset: (x: 9pt, y: 8pt), radius: (bottom-left: 2pt, bottom-right: 2pt), { @@ -320,6 +466,26 @@ it }, ) + // DEVIATION (print): a line of code breaks only at spaces and a few marks, so + // a long URL or identifier ran past the frame. A zero-width space after + // / . _ - & ; = ? , lets it wrap inside; the spaces would end up in copied + // code, which paper does not have. They go into the block's text: a regex + // show rule exceeds Typst's grouping depth on a long block. Defined after the + // frame, so it runs first and the frame takes the new block. A block without + // a mark to break after, or with a zero-width space already, is left as it + // is: every block the rule replaces holds one, which ends the recursion. + // Terminal tables keep their lines: codeblock() shrinks them to fit. + show raw.where(block: true): it => if ( + not print-edition + or it.text.contains(sym.zws) + or not it.text.contains(regex("[/._&;=?,-]")) + or it.text.contains(regex("[─-╿]")) + ) { it } else { + let fields = it.fields() + let _ = fields.remove("text") + let _ = fields.remove("lines", default: none) + raw(..fields, it.text.replace(regex("[/._&;=?,-]"), mark => mark.text + sym.zws)) + } // -- headings ------------------------------------------------------------------- // DEVIATION: one level deeper than a policy. A navigation section is a chapter @@ -370,6 +536,33 @@ show heading.where(level: 6): it => block(above: 1.3em, below: 0.7em, sticky: true, text(size: size-h3, weight: "medium", fill: ec-grey, it.body)) + // -- figures ---------------------------------------------------------------------- + // DEVIATION: image figures carry no number. The site numbers none, and pages + // that need numbers write them into the caption and refer to them in the text + // ("see figure 3"); a book-wide number in front printed "Figure 15: Figure 2:". + show figure.where(kind: image): set figure(numbering: none) + // DEVIATION: a figure does not break across pages, and Typst does not shrink an + // image taller than the page, so a tall screenshot at the column width ran into + // the footer. An image in a figure that is taller than the page body, less room + // for a two-line caption, is scaled down to that height, in proportion. Only + // images in figures: `layout` is block-level, and an image in a line of text + // stays inline. The scaled image has a height, which ends the recursion. + show figure.where(kind: image): it => { + show image: img => if img.height != auto { img } else { + layout(region => { + let room = region.height - 4em.to-absolute() + if measure(img, width: region.width).height <= room { img } else { + let fields = img.fields() + let source = fields.remove("source") + let _ = fields.remove("width", default: none) + let _ = fields.remove("height", default: none) + image(source, ..fields, height: room) + } + }) + } + it + } + // -- tables: booktabs (house style) ----------------------------------------------- set table( inset: (x: 5pt, y: 5pt), @@ -404,7 +597,16 @@ // DEVIATION: a page of its own. A policy runs from its title block straight // into the text; a manual of this length gets a title page, and its contents // start on the next one. The title block itself is the house style's. - v(3.2cm) + // + // DEVIATION (print): page 1 of the book block. The header that carries the + // logo on screen is empty in print, so the logo sits on the page itself. The + // page names the publisher; date, link and copyright move to the imprint. + if print-edition { + place(top + left, dx: -28pt, dy: -1.3cm, image("logo.svg", width: 4cm)) + v(4.6cm) + } else { + v(3.2cm) + } text(size: size-eyebrow, fill: ec-grey, weight: "regular", tracking: 0.06em, upper(context-line)) v(0.35em, weak: true) block( @@ -418,17 +620,31 @@ align(right, text(size: size-subtitle, weight: "light", fill: ec-slate, subtitle)) v(0.5em, weak: true) line(length: 100%, stroke: 0.5pt + ec-grey.lighten(50%)) - v(0.5em, weak: true) - align(right, text(size: size-meta, fill: ec-grey, weight: "regular")[Generated #generated]) - v(1fr) - block(text(size: size-meta, fill: ec-grey, weight: "regular", { - set par(leading: 0.56em, justify: false) - link(site-url) - linebreak() - copyright - })) + if print-edition { + v(1fr) + text(size: size-subtitle, weight: "regular", fill: ec-slate, yaml("print.yml").publisher.name) + } else { + v(0.5em, weak: true) + align(right, text(size: size-meta, fill: ec-grey, weight: "regular")[Generated #generated]) + v(1fr) + block(text(size: size-meta, fill: ec-grey, weight: "regular", { + set par(leading: 0.56em, justify: false) + link(site-url) + linebreak() + copyright + })) + } pagebreak() + // -- imprint (print) ------------------------------------------------------------------ + if print-edition { + imprint( + title: title, context-line: context-line, subtitle: subtitle, + generated: generated, site-url: site-url, copyright: copyright, + ) + pagebreak() + } + // -- contents ---------------------------------------------------------------------- // The parts only. Each part opens with the contents of its pages. contents-title @@ -439,7 +655,21 @@ show outline.entry: set block(above: 1.1em) outline(title: none, depth: 1, indent: auto) } - pagebreak(weak: true) + // In print the break before the first part follows directly: a weak break here + // would leave the blank page before part A outside its markers. + if not print-edition { + pagebreak(weak: true) + } body + + // DEVIATION (print): the build reads the page count from this marker and asks + // for one blank last page when it is odd (tools/build_pdf.py, unpadded_pages). + if print-edition { + context [#metadata(here().page()) ] + if sys.inputs.at("pad", default: "false") == "true" { + pagebreak() + [#metadata("pad") ] + } + } } diff --git a/tools/pdf_authors.py b/tools/pdf_authors.py new file mode 100644 index 000000000..0e72826b3 --- /dev/null +++ b/tools/pdf_authors.py @@ -0,0 +1,104 @@ +"""Write the author list the imprint of the print edition prints. + +The authors are the GitHub accounts that contributed to the documentation +repository, most commits first. Anonymous contributions - commits whose e-mail +maps to no GitHub account - bot accounts and software agents are not listed. + +The list is committed as tools/pdf/authors.yml, so a PDF build stays offline +and reproducible, and a changed list shows up in review. Invoked by +`task pdf:authors`; GITHUB_TOKEN or GH_TOKEN is used when set. +""" +from __future__ import annotations + +import json +import os +import re +import urllib.request +from pathlib import Path + +import click +import yaml + +REPOSITORY = "eccenca/documentation.eccenca.com" +AUTHORS_YML = Path("tools/pdf/authors.yml") +API = "https://api.github.com" +# Agents commit under ordinary accounts of type User, so they are named here. +AGENT = re.compile(r"claude|codex", re.IGNORECASE) + + +def select_authors(contributors: list[dict]) -> list[dict]: + """The contributors the imprint lists, in its order: most commits first, ties by ID, case-insensitive.""" + people = [ + c for c in contributors + if c.get("type") == "User" and c.get("login") and not AGENT.search(c["login"]) + ] + people.sort(key=lambda c: (-int(c["contributions"]), c["login"].casefold())) + return [{"id": c["login"], "commits": int(c["contributions"])} for c in people] + + +def next_page(link_header: str) -> str | None: + """The `rel="next"` URL of a GitHub Link header, if there is one.""" + match = re.search(r'<([^>]+)>;\s*rel="next"', link_header) + return match.group(1) if match else None + + +def fetch_contributors(repository: str, token: str | None) -> list[dict]: + """Every contributor of a repository, following the API's pagination. + + Without `anon=1` the API leaves out contributions that map to no account. + """ + url: str | None = f"{API}/repos/{repository}/contributors?per_page=100" + contributors: list[dict] = [] + while url: + request = urllib.request.Request(url, headers={"Accept": "application/vnd.github+json"}) + if token: + request.add_header("Authorization", f"Bearer {token}") + with urllib.request.urlopen(request, timeout=30) as response: + contributors.extend(json.load(response)) + url = next_page(response.headers.get("Link", "")) + return contributors + + +class IndentedDumper(yaml.SafeDumper): + """Indents list items under their key, as the repository's yamllint rules require.""" + + def increase_indent(self, flow: bool = False, indentless: bool = False) -> None: + return super().increase_indent(flow, False) + + +def authors_yaml(repository: str, authors: list[dict]) -> str: + """The committed file: a header naming its source, then the list in imprint order.""" + header = ( + "---\n" + "# Generated by `dec-tool pdf-authors` from the GitHub contributors of\n" + f"# {repository}. Do not edit - run `task pdf:authors`.\n" + ) + body = yaml.dump( + {"repository": repository, "authors": authors}, + Dumper=IndentedDumper, sort_keys=False, allow_unicode=True, + ) + return header + body + + +@click.command(name="pdf-authors") +@click.option( + "--repository", + default=REPOSITORY, + help="Which GitHub repository's contributors are the authors?", + show_default=True, +) +@click.option( + "--output-file", "-o", + type=click.Path(exists=False, dir_okay=False, file_okay=True), + default=str(AUTHORS_YML), + help="Where to write the author list?", + show_default=True, +) +def pdf_authors(repository: str, output_file: str) -> None: + """Write the imprint's author list from the repository's GitHub contributors.""" + token = os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN") + authors = select_authors(fetch_contributors(repository, token)) + if not authors: + raise click.ClickException(f"no contributors found for {repository}") + Path(output_file).write_text(authors_yaml(repository, authors), encoding="utf-8") + print(f"{len(authors)} authors written to {output_file}") diff --git a/tools/update_di_reference.py b/tools/update_di_reference.py index 6894509e0..b4f75187b 100644 --- a/tools/update_di_reference.py +++ b/tools/update_di_reference.py @@ -25,8 +25,20 @@ undefined=StrictUndefined ) +# A fenced block, with or without a language. Squeezed onto one line it is no +# fence, but rumdl --fix still takes it for one and inserts a blank line before +# it - which ends the operator table whose row the description is. +FENCED_BLOCK = re.compile(r"```(?:\w*\n)?(.*?)```", re.DOTALL) + + +def code_span(match: re.Match) -> str: + """A fenced block's content as a code span, delimited by two backticks when it holds one.""" + content = re.sub(r"\s+", " ", match.group(1)).strip() + return f"`` {content} ``" if "`" in content else f"`{content}`" + + def stripped_single_line(value: str) -> str: - return re.sub(r"\s+", " ", value).strip() + return re.sub(r"\s+", " ", FENCED_BLOCK.sub(code_span, value)).strip() class PluginReference(BaseModel): """Reference to a related plugin.""" From f344992954c575ab8618d0bd4fe62bbeb1e09a15 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Tue, 15 Sep 2026 12:24:43 +0200 Subject: [PATCH 02/15] print author names instead of GitHub IDs in the imprint The imprint listed contributors by their GitHub IDs. - task pdf:authors stores each profile name in tools/pdf/authors.yml and adds new authors to authors.names in tools/pdf/print.yml - print.yml names authors (with titles) and excludes IDs; excluded names are never looked up - the build prints the given name, else the profile name, else the ID, and warns about each ID it prints - use the GitHub CLI token when no token is set, and end a rate limit, a rejected token or a network error with a message --- Taskfile.yml | 19 ++- tasks/backlog.md | 20 ++- tasks/spec.md | 20 ++- tests/test_pdf_authors.py | 262 ++++++++++++++++++++++++++++++++++- tools/README.md | 3 +- tools/build_pdf.py | 12 ++ tools/pdf/authors.yml | 28 +++- tools/pdf/print.yml | 38 +++++- tools/pdf/style.typ | 11 +- tools/pdf/template.typ | 1 + tools/pdf_authors.py | 277 +++++++++++++++++++++++++++++++++++--- 11 files changed, 640 insertions(+), 51 deletions(-) diff --git a/Taskfile.yml b/Taskfile.yml index 93956aecf..ad056052a 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -210,10 +210,10 @@ tasks: imprint, links printed as page references and footnotes, and images normalized to 300 ppi without transparency. - tools/pdf/print.yml configures the edition: the publisher, and how each - navigation section is printed (full, list or omit). - tools/pdf/authors.yml is the imprint's author list; refresh it with - `task pdf:authors`. + tools/pdf/print.yml configures the edition: the publisher, the authors' + names and exclusions, and how each navigation section is printed (full, + list or omit). tools/pdf/authors.yml is the imprint's author list; + refresh it with `task pdf:authors`. Writes dist/documentation-eccenca-com--print.pdf; the merged HTML, the Typst source and the normalized images stay in dist/pdf/print/. The @@ -229,9 +229,14 @@ tasks: desc: Refresh the author list of the print edition's imprint summary: | Writes tools/pdf/authors.yml from the GitHub contributors of this - repository: most commits first, without anonymous contributions, bot - accounts and software agents. Commit the result. Uses GITHUB_TOKEN or - GH_TOKEN when set; unauthenticated requests are rate limited. + repository and the names their GitHub profiles show: most commits + first, without anonymous contributions, bot accounts, software agents + and the IDs excluded in tools/pdf/print.yml. Adds each author that + print.yml does not list yet to its authors.names, without a name, and + warns about each author who would print as a GitHub ID. Commit both + files. Uses GITHUB_TOKEN or GH_TOKEN, else the token of a logged-in + GitHub CLI (`gh auth login`); without a token GitHub allows 60 + requests an hour, and a run takes one per author. deps: - task: install cmds: diff --git a/tasks/backlog.md b/tasks/backlog.md index a0e658bb7..33c3ed484 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -10,7 +10,7 @@ Every task names how it is verified. A task is not done until that verification ## P0 - Decisions - done -Spec §4, D1-D11: BoD, A4, black and white on 80 g, no ISBN, GitHub IDs as authors, separate screen and +Spec §4, D1-D11: BoD, A4, black and white on 80 g, no ISBN, authors by name (first GitHub IDs, revised 2026-09-15), separate screen and print editions, section modes with A.3 and Release Notes as lists, page references and URL footnotes, no logo or version on text pages, 10 pt body, optional Ghostscript normalization. @@ -119,6 +119,21 @@ exclusion of bots, agents and anonymous entries, pagination, file format), `task generated `tools/pdf/authors.yml` lists the 22 IDs of spec §3 in the same order and passes yamllint - list items are indented, which PyYAML's default dumper does not do. +**Revised 2026-09-15 - names instead of IDs (D4):** `dec-tool pdf-authors` also writes the name each +GitHub profile shows (`GET /users/`), and skips the IDs in `authors.exclude` of `tools/pdf/print.yml` +before any lookup. The build applies `authors.names` - for a profile without a name, or to add a title - +and the exclusions to the committed list (`load_imprint_names`), passes the names to Typst as the +`authors` input, and warns about each author it still prints as an ID. 16 of the 22 profiles show a +name; `rpietzsch`, `annamakor`, `MaximilianWenzel`, `adelahaye-ecc`, `dgrtner-ecc` and `pkgut` need an +entry. Each run also adds the authors `authors.names` does not list yet, without a name and with the +profile's name in a comment (`prefill_names`); it edits `print.yml` as text, so comments stay, and reads +it back to check that only those entries changed. A name left empty prints the profile's. First run: all +22 IDs added; a second run adds none. Requests use GITHUB_TOKEN or GH_TOKEN, else the token of a +logged-in GitHub CLI; without one GitHub allows 60 requests an hour, and a run takes one per author. A +failed request - rate limit with its reset time, rejected token, unreachable API - ends the command with +a message instead of a traceback, before any file is written. `tests/test_pdf_authors.py` has 28 tests: +rules, exclusion, profile names, the order of names, the prefill, token sources and API failures. + --- ## P7 - Imprint - **done** @@ -136,6 +151,9 @@ edition - with a sentence that the print edition shortens sections whenever `pri `list` or `omit` - and the typesetting. Checked on the rendered page and in its text: the author IDs match `authors.yml` in order. The licence URL is set as a string, because Typst links URLs written in markup. +**Revised 2026-09-15:** the imprint prints the authors' names, not their IDs. `imprint()` no longer +reads `authors.yml`; the build passes the names as the `authors` input (P6). + --- ## P8 - Section modes - **done** diff --git a/tasks/spec.md b/tasks/spec.md index 843af395d..9d5e3a5d8 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -101,11 +101,15 @@ Page 2, verso, no page furniture. Contents: #### Authors -Rule (D4): GitHub accounts of the contributors to +Rule (D4, revised 2026-09-15): GitHub accounts of the contributors to [eccenca/documentation.eccenca.com](https://github.com/eccenca/documentation.eccenca.com), **most -commits first**, printed as their **GitHub IDs**. Not listed: anonymous contributions (commits whose -e-mail maps to no GitHub account), bot accounts and software agents (codex, claude). Ties are ordered by -ID, case-insensitive. +commits first**, printed with their **names**: the name in the hand-maintained `authors.names` of +`tools/pdf/print.yml` - for a profile without a name, or to add a title - else the name the GitHub +profile shows, else the GitHub ID. Not listed: anonymous contributions (commits whose e-mail maps to no +GitHub account), bot accounts, software agents (codex, claude) and the IDs in `authors.exclude`, whose +names are not looked up. Ties are ordered by ID, case-insensitive. `task pdf:authors` adds each author +that `authors.names` does not list yet, without a name and with the profile's name in a comment, so all +names are maintained in one place; an entry left empty prints the profile's name. Measured with `gh api repos/eccenca/documentation.eccenca.com/contributors` on 2026-09-14: @@ -136,6 +140,10 @@ Measured with `gh api repos/eccenca/documentation.eccenca.com/contributors` on 2 Dropped: two anonymous entries (33 and 2 commits). +Names, checked with `gh api users/` on 2026-09-15: 16 of the 22 profiles show a name. `rpietzsch`, +`annamakor`, `MaximilianWenzel`, `adelahaye-ecc`, `dgrtner-ecc` and `pkgut` show none; they need an entry +in `authors.names`, or they print as their ID, and the build warns about each. + - **No software agent is in the list today.** Claude's commits carry a `Co-Authored-By` trailer and count for the human author; GitHub does not list co-authors as contributors. The exclusion rule still belongs in the tooling. @@ -156,7 +164,7 @@ All made 2026-09-14. | D1 | Scope | The print edition reduces sections instead of printing them in full, **configurable per section** as `full`, `list` or `omit`. Default: A.3 Task and Operator Reference → `list`, H Release Notes → `list`, everything else `full` (§5) | | D2 | Provider | **BoD** (§2) | | D3 | ISBN | none for now; may come later | -| D4 | Authors | GitHub IDs, most commits first, no anonymous entries, no bots or agents (§3) | +| D4 | Authors | names, most commits first: from `authors.names` in `print.yml`, else the GitHub profile, else the ID; no anonymous entries, bots, agents or excluded IDs (§3). Revised 2026-09-15 - first decided as GitHub IDs | | D5 | Colour and paper | black and white interior on 80 g paper. BoD offers 80 g to publishers only - eccenca GmbH needs a publisher account | | D6 | Editions | separate screen and print PDFs. `task pdf` stays the screen edition, unchanged; `task pdf:print` builds the book block, switched by a Typst input (`edition=print`) and a print configuration | | D7 | Trim size | A4 | @@ -347,7 +355,7 @@ the same task, for checking the palette (P11). - `task pdf` builds the screen PDF exactly as before: 1680 pages, unchanged look - `task pdf:print` builds the book block with the configuration in `tools/pdf/print.yml` - page 1 is the title page naming eccenca GmbH as publisher; page 2 is the imprint with the publisher's - address and the authors from `tools/pdf/authors.yml`, most commits first + address and the authors by name, from `tools/pdf/authors.yml` and `print.yml`, most commits first - on every numbered page the page number sits at the outer edge: left on even pages, right on odd pages - checked by the preflight report, not by eye - title page, imprint, part covers and blank pages carry no header and no footer; text pages carry no diff --git a/tests/test_pdf_authors.py b/tests/test_pdf_authors.py index ec5046811..2ada8ce97 100644 --- a/tests/test_pdf_authors.py +++ b/tests/test_pdf_authors.py @@ -1,7 +1,50 @@ """Test the author list of the print edition's imprint""" +import shutil +import subprocess +import time +import urllib.error +import urllib.request + +import click +import pytest import yaml -from tools.pdf_authors import authors_yaml, next_page, select_authors +from tools.pdf_authors import ( + AuthorRules, + add_names, + authors_yaml, + get_json, + github_token, + imprint_names, + load_author_rules, + load_imprint_names, + next_page, + prefill_names, + select_authors, +) + +# The shape of tools/pdf/print.yml: comments, and sections before and after `authors`. +PRINT_YML = """\ +--- +# The print edition. +publisher: + name: eccenca GmbH + +authors: + # GitHub ID: the name to print. + names: {} + # GitHub IDs never to list. + exclude: [] + +sections: + build/reference/: list +""" + +AUTHORS = [ + {"id": "seebi", "name": "Jane Doe", "commits": 3}, + {"id": "nameless", "name": None, "commits": 2}, + {"id": "Hubot", "name": "John Roe", "commits": 1}, +] def test_most_commits_first_and_ties_by_id_case_insensitive(): @@ -25,6 +68,25 @@ def test_bots_agents_and_anonymous_contributions_are_not_authors(): assert select_authors(contributors) == [{"id": "sobo", "commits": 53}] +def test_excluded_ids_are_not_authors_whatever_their_case(): + contributors = [ + {"login": "seebi", "contributions": 532, "type": "User"}, + {"login": "BorderCloud", "contributions": 10, "type": "User"}, + ] + rules = AuthorRules(names={}, exclude=frozenset({"bordercloud"})) + assert select_authors(contributors, rules) == [{"id": "seebi", "commits": 532}] + + +def test_names_are_the_profile_names_and_a_blank_one_is_none(): + authors = [{"id": "a", "commits": 3}, {"id": "b", "commits": 2}, {"id": "c", "commits": 1}] + profiles = {"a": " Jane Doe ", "b": "", "c": None} + assert add_names(authors, profiles.get) == [ + {"id": "a", "name": "Jane Doe", "commits": 3}, + {"id": "b", "name": None, "commits": 2}, + {"id": "c", "name": None, "commits": 1}, + ] + + def test_pagination_follows_the_next_link_only(): header = ( '; rel="next", ' @@ -35,11 +97,197 @@ def test_pagination_follows_the_next_link_only(): def test_the_file_names_its_source_and_keeps_the_order(): - text = authors_yaml("eccenca/docs", [{"id": "b", "commits": 2}, {"id": "a", "commits": 1}]) + authors = [{"id": "b", "name": "Jane Doe", "commits": 2}, {"id": "a", "name": None, "commits": 1}] + text = authors_yaml("eccenca/docs", authors) assert text.startswith("---\n# Generated by `dec-tool pdf-authors`") # yamllint's default rules want list items indented under their key. - assert "authors:\n - id: b\n commits: 2\n" in text - assert yaml.safe_load(text) == { - "repository": "eccenca/docs", - "authors": [{"id": "b", "commits": 2}, {"id": "a", "commits": 1}], - } + assert "authors:\n - id: b\n name: Jane Doe\n commits: 2\n" in text + assert yaml.safe_load(text) == {"repository": "eccenca/docs", "authors": authors} + + +def test_rules_read_names_and_exclusions_case_insensitive(tmp_path): + path = tmp_path / "print.yml" + path.write_text("authors:\n names:\n Octocat: Dr. Jane Doe\n exclude:\n - Hubot\n") + assert load_author_rules(path) == AuthorRules(names={"octocat": "Dr. Jane Doe"}, exclude=frozenset({"hubot"})) + + +def test_without_an_authors_section_nothing_is_renamed_or_excluded(tmp_path): + path = tmp_path / "print.yml" + path.write_text("publisher:\n name: eccenca GmbH\n") + assert load_author_rules(path) == AuthorRules(names={}, exclude=frozenset()) + + +@pytest.mark.parametrize( + "section", + [ + "authors:\n exclude: hubot\n", + "authors:\n names:\n - octocat\n", + "authors:\n names:\n octocat: 42\n", + ], +) +def test_malformed_rules_fail(tmp_path, section): + path = tmp_path / "print.yml" + path.write_text(section) + with pytest.raises(click.ClickException, match="authors"): + load_author_rules(path) + + +def test_a_name_left_empty_is_not_given(tmp_path): + path = tmp_path / "print.yml" + path.write_text("authors:\n names:\n octocat:\n hubot: ''\n") + assert load_author_rules(path) == AuthorRules(names={}, exclude=frozenset()) + + +def test_prefill_lists_every_author_with_the_profile_name_as_a_comment(): + text, added = prefill_names(PRINT_YML, AUTHORS) + assert added == ["seebi", "nameless", "Hubot"] + assert ( + " # GitHub ID: the name to print.\n" + " names:\n" + " seebi: # GitHub profile: Jane Doe\n" + " nameless: # no name on the GitHub profile\n" + " Hubot: # GitHub profile: John Roe\n" + " # GitHub IDs never to list.\n" + " exclude: []\n" + ) in text + data = yaml.safe_load(text) + # Left empty, an entry gives no name: the imprint keeps printing the profile's. + assert data["authors"] == {"names": {"seebi": None, "nameless": None, "Hubot": None}, "exclude": []} + assert data["publisher"] == {"name": "eccenca GmbH"} + assert data["sections"] == {"build/reference/": "list"} + assert text.startswith("---\n# The print edition.\n") + + +def test_prefill_keeps_the_names_given_and_adds_only_the_ids_not_listed(tmp_path): + given = PRINT_YML.replace(" names: {}\n", " names:\n SEEBI: Dr. Jane Doe\n") + text, added = prefill_names(given, AUTHORS) + assert added == ["nameless", "Hubot"] + assert ( + " names:\n" + " SEEBI: Dr. Jane Doe\n" + " nameless: # no name on the GitHub profile\n" + " Hubot: # GitHub profile: John Roe\n" + " # GitHub IDs never to list.\n" + ) in text + path = tmp_path / "print.yml" + path.write_text(text) + assert load_author_rules(path) == AuthorRules(names={"seebi": "Dr. Jane Doe"}, exclude=frozenset()) + + +def test_prefill_skips_excluded_ids_and_a_second_run_changes_nothing(): + excluding = PRINT_YML.replace(" exclude: []\n", " exclude:\n - hubot\n") + once, added = prefill_names(excluding, AUTHORS) + assert added == ["seebi", "nameless"] + assert prefill_names(once, AUTHORS) == (once, []) + + +def test_prefill_adds_an_authors_section_where_there_is_none(): + text, added = prefill_names("---\npublisher:\n name: eccenca GmbH\n", [{"id": "null", "name": None, "commits": 1}]) + assert added == ["null"] + # A login YAML would read as something else is quoted. + assert "\nauthors:\n names:\n 'null': # no name on the GitHub profile\n exclude: []\n" in text + assert yaml.safe_load(text) == {"publisher": {"name": "eccenca GmbH"}, "authors": {"names": {"null": None}, "exclude": []}} + + +def failing_urlopen(error): + def urlopen(request, timeout): + raise error + return urlopen + + +def rate_limited(reset: int) -> urllib.error.HTTPError: + headers = {"X-RateLimit-Remaining": "0", "X-RateLimit-Reset": str(reset)} + return urllib.error.HTTPError("https://api.github.com/users/x", 403, "rate limit exceeded", headers, None) + + +def test_a_rate_limit_fails_with_the_reset_time_and_how_to_raise_the_limit(monkeypatch): + reset = 1_900_000_000 + monkeypatch.setattr(urllib.request, "urlopen", failing_urlopen(rate_limited(reset))) + with pytest.raises(click.ClickException) as raised: + get_json("https://api.github.com/users/x", None) + message = raised.value.message + assert "rate limit" in message + assert time.strftime("%H:%M", time.localtime(reset)) in message + assert "GITHUB_TOKEN" in message and "gh auth login" in message + + +def test_with_a_token_a_rate_limit_fails_with_the_reset_time_only(monkeypatch): + monkeypatch.setattr(urllib.request, "urlopen", failing_urlopen(rate_limited(1_900_000_000))) + with pytest.raises(click.ClickException) as raised: + get_json("https://api.github.com/users/x", "a-token") + assert "rate limit" in raised.value.message + assert "GITHUB_TOKEN" not in raised.value.message + + +@pytest.mark.parametrize( + "error, expected", + [ + (urllib.error.HTTPError("https://api.github.com/users/gone", 404, "Not Found", {}, None), "HTTP 404 Not Found"), + (urllib.error.HTTPError("https://api.github.com/users/x", 401, "Unauthorized", {}, None), "rejected the token"), + (urllib.error.URLError("nodename nor servname provided"), "cannot reach the GitHub API"), + (TimeoutError("timed out"), "cannot reach the GitHub API"), + ], +) +def test_other_failures_end_with_a_message_instead_of_a_traceback(monkeypatch, error, expected): + monkeypatch.setattr(urllib.request, "urlopen", failing_urlopen(error)) + with pytest.raises(click.ClickException, match=expected): + get_json("https://api.github.com/users/gone", "a-token") + + +def without_token_variables(monkeypatch): + monkeypatch.delenv("GITHUB_TOKEN", raising=False) + monkeypatch.delenv("GH_TOKEN", raising=False) + + +def test_the_token_of_the_environment_comes_first(monkeypatch): + without_token_variables(monkeypatch) + monkeypatch.setenv("GH_TOKEN", "from-env") + monkeypatch.setattr(shutil, "which", lambda name: pytest.fail("the GitHub CLI is not asked")) + assert github_token() == ("from-env", "GH_TOKEN") + + +def test_without_one_in_the_environment_the_github_cli_lends_its_token(monkeypatch): + without_token_variables(monkeypatch) + monkeypatch.setattr(shutil, "which", lambda name: "/usr/local/bin/gh") + monkeypatch.setattr( + subprocess, "run", lambda *args, **kwargs: subprocess.CompletedProcess(args, 0, stdout="gho_cli\n", stderr="") + ) + assert github_token() == ("gho_cli", "the GitHub CLI") + + +@pytest.mark.parametrize( + "which, run", + [ + (lambda name: None, None), + ( + lambda name: "/usr/local/bin/gh", + lambda *args, **kwargs: subprocess.CompletedProcess(args, 1, stdout="", stderr="not logged in"), + ), + ], +) +def test_without_a_variable_or_a_logged_in_github_cli_there_is_no_token(monkeypatch, which, run): + without_token_variables(monkeypatch) + monkeypatch.setattr(shutil, "which", which) + if run is not None: + monkeypatch.setattr(subprocess, "run", run) + assert github_token() == (None, None) + + +def test_the_imprint_prints_a_given_name_before_the_profile_name_and_the_id_last(): + authors = [ + {"id": "a", "name": "Jane Doe", "commits": 4}, + {"id": "B", "name": "John Roe", "commits": 3}, + {"id": "c", "name": None, "commits": 2}, + {"id": "d", "name": "Richard Miles", "commits": 1}, + ] + rules = AuthorRules(names={"b": "Dr. John Roe"}, exclude=frozenset({"d"})) + # The names in imprint order, and the IDs that print as themselves for want of a name. + assert imprint_names(authors, rules) == (["Jane Doe", "Dr. John Roe", "c"], ["c"]) + + +def test_the_imprint_reads_the_generated_list_and_the_rules(tmp_path): + authors_yml = tmp_path / "authors.yml" + authors_yml.write_text(authors_yaml("eccenca/docs", [{"id": "a", "name": None, "commits": 1}])) + print_yml = tmp_path / "print.yml" + print_yml.write_text("authors:\n names:\n a: Jane Doe\n") + assert load_imprint_names(authors_yml, print_yml) == (["Jane Doe"], []) diff --git a/tools/README.md b/tools/README.md index 5b77b8406..de9df3d43 100644 --- a/tools/README.md +++ b/tools/README.md @@ -10,7 +10,8 @@ supported entry points and pass the right options. | Command | Used by | Purpose | | :------ | :------ | :------ | | `build-navigation` | `task update:navigation`, `task check:navigation` | Build `nav.yml` from the `docs/**/.pages` files; `--check` diffs instead of writing and fails on drift | -| `build-pdf` | `task pdf` | Merge the built `site/` along `nav.yml` into one document and typeset it as a single PDF with pandoc and Typst; style, filter and fonts are in `pdf/` | +| `build-pdf` | `task pdf`, `task pdf:print` | Merge the built `site/` along `nav.yml` into one document and typeset it as a single PDF with pandoc and Typst; style, filter and fonts are in `pdf/` | +| `pdf-authors` | `task pdf:authors` | Write the print edition's author list `pdf/authors.yml` from the GitHub contributors and the names their profiles show; names and exclusions are kept in `pdf/print.yml`, where it adds each new author without a name | | `check-zensical-output` | `task check:output` | Inspect the built `site/` and fail if a feature we reimplemented for Zensical regressed | | `localize-bundle-assets` | `task build` | Rewrite the third-party asset URLs Zensical bakes into its JavaScript bundle to the vendored copies | | `update-icons` | `task update:icons` | Fetch the eccenca icon set from the gui-elements repository | diff --git a/tools/build_pdf.py b/tools/build_pdf.py index 2d2ba7619..371924e4c 100644 --- a/tools/build_pdf.py +++ b/tools/build_pdf.py @@ -38,6 +38,8 @@ from bs4 import BeautifulSoup, NavigableString from PIL import Image +from tools.pdf_authors import load_imprint_names + DEFAULT_OUT_STEM = "dist/documentation-eccenca-com" SITE_DIR = Path("site") NAV_YML = Path("nav.yml") @@ -1176,6 +1178,16 @@ def build_pdf( # call is the one it always was. if print_edition: inputs["edition"] = "print" + # The imprint's authors: the committed list with the names and + # exclusions of print.yml applied. + names, unnamed = load_imprint_names() + if unnamed: + print( + f"WARNING: the imprint prints {len(unnamed)} authors as their GitHub ID, for want of a name " + f"in {PRINT_YML} (authors.names): {', '.join(unnamed)}", + file=sys.stderr, + ) + inputs["authors"] = ", ".join(names) for key, value in inputs.items(): command += ["--input", f"{key}={value}"] if print_edition and unpadded_pages(command, typ_path) % 2: diff --git a/tools/pdf/authors.yml b/tools/pdf/authors.yml index 7ff2db061..fb39531c7 100644 --- a/tools/pdf/authors.yml +++ b/tools/pdf/authors.yml @@ -1,49 +1,63 @@ --- # Generated by `dec-tool pdf-authors` from the GitHub contributors of -# eccenca/documentation.eccenca.com. Do not edit - run `task pdf:authors`. +# eccenca/documentation.eccenca.com and the names their profiles show. +# Do not edit - run `task pdf:authors`. Names and exclusions: tools/pdf/print.yml. repository: eccenca/documentation.eccenca.com authors: - id: rpietzsch + name: null commits: 639 - id: seebi + name: Sebastian Tramp commits: 532 - id: irangareddy + name: Ranga Reddy commits: 66 - id: muddymudskipper + name: Thomas commits: 62 - id: mgns + name: Magnus commits: 61 - id: sobo + name: Thomas Telleis commits: 53 - id: annamakor + name: null commits: 29 - - id: msaipraneeth - commits: 25 - - id: saradaimi786 - commits: 22 - id: louiswesterheide + name: Louis commits: 20 - id: robertisele + name: Robert Isele commits: 18 - id: spl0tt + name: Sebastian Plott commits: 12 - id: white-gecko + name: Natanael Arndt commits: 11 - id: BorderCloud + name: Karima Rafes commits: 10 - id: haschek + name: Michael Haschke commits: 10 - id: MaximilianWenzel + name: null commits: 9 - id: tomatophantastico + name: Jörg Unbehauen commits: 7 - id: adelahaye-ecc + name: null commits: 5 - id: dgrtner-ecc + name: null commits: 1 - id: looooph + name: Istvan Mocsy commits: 1 - id: peterfreytag - commits: 1 - - id: pkgut + name: Peter Freytag commits: 1 diff --git a/tools/pdf/print.yml b/tools/pdf/print.yml index 3b6c17c5a..20c0b6e40 100644 --- a/tools/pdf/print.yml +++ b/tools/pdf/print.yml @@ -1,6 +1,7 @@ --- # The print edition of the PDF: `task pdf:print`, see tasks/spec.md. -# tools/build_pdf.py reads `sections`; tools/pdf/style.typ reads the rest. +# tools/build_pdf.py reads `sections`, tools/pdf_authors.py `authors`, and +# tools/pdf/style.typ the rest. # Printed on the title page and in the imprint. publisher: @@ -10,6 +11,41 @@ publisher: - 04109 Leipzig - Germany +# The imprint's authors. `task pdf:authors` writes tools/pdf/authors.yml from +# the GitHub contributors, most commits first, with the name each GitHub +# profile shows. The imprint prints the name given here, else the profile's, +# else the GitHub ID - the build warns about each author it prints as an ID. +authors: + # GitHub ID: the name to print - for a profile that shows no name, or to add + # a title. IDs match case-insensitively. `task pdf:authors` adds each new + # author without a name and the profile's name in a comment; an entry left + # empty prints the profile's name. + names: + rpietzsch: René Pietzsch + seebi: Dr. Sebastian Tramp + irangareddy: Ranga Reddy + muddymudskipper: Dr. Thomas Wilmering + mgns: Magnus Knuth + sobo: Thomas Telleis + annamakor: Annamária Koren + msaipraneeth: # GitHub profile: Saipraneeth + saradaimi786: # GitHub profile: syeda sara daimi + louiswesterheide: Louis Westerheide + robertisele: Dr. Robert Isele + spl0tt: Sebastian Plott + white-gecko: Dr. Natanael Arndt + BorderCloud: Dr. Karima Rafes + haschek: Michael Haschke + MaximilianWenzel: Maximilian Wenzel + tomatophantastico: Jörg Unbehauen + adelahaye-ecc: Alexis Delahaye + dgrtner-ecc: Denis Gärtner + looooph: Istvan Mocsy + peterfreytag: Peter Freytag + pkgut: # no name on the GitHub profile + # GitHub IDs never to list; `task pdf:authors` does not look up their names. + exclude: [msaipraneeth, saradaimi786, pkgut] + # The weight of the body text: light, as on screen, or regular, which prints # darker in black and white. body-weight: light diff --git a/tools/pdf/style.typ b/tools/pdf/style.typ index 03e07897c..9737b831b 100644 --- a/tools/pdf/style.typ +++ b/tools/pdf/style.typ @@ -266,19 +266,19 @@ // DEVIATION (print): the back of the title page is the imprint - edition, // publisher, authors, licence and the online edition - and it carries the // edition stamp the screen PDF prints in its header. tools/pdf/print.yml holds -// the publisher and the section modes, tools/pdf/authors.yml the authors. -#let imprint(title: "", context-line: "", subtitle: "", generated: "", site-url: "", copyright: "") = { +// the publisher and the section modes. The authors arrive as an input: the build +// applies the names and exclusions of print.yml to tools/pdf/authors.yml. +#let imprint(title: "", context-line: "", subtitle: "", generated: "", site-url: "", copyright: "", authors: "") = { let config = yaml("print.yml") let mode(section) = if type(section) == dictionary { section.at("mode", default: "full") } else { section } let shortened = config.at("sections", default: (:)).values().any(section => mode(section) != "full") - let authors = yaml("authors.yml").authors.map(author => author.id) set text(size: size-small, fill: ec-slate) set par(justify: false, leading: 0.6em, spacing: 1.2em) let entry(label, body) = block(below: 1.3em, [#text(weight: "bold", label) \ #body]) v(1fr) entry(title)[#context-line, #subtitle \ Print edition, generated #generated] entry("Publisher")[#config.publisher.name \ #config.publisher.address.join(linebreak())] - entry("Authors")[The contributors to the documentation by their GitHub IDs, most commits first: #authors.join(", ").] + entry("Authors")[The contributors to the documentation, most commits first: #authors.] // The URL is a string: written as markup, Typst would turn it into a link. entry("Licence")[ This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License, @@ -306,6 +306,7 @@ version: "", site-url: "", copyright: "", + authors: "", body, ) = { set document( @@ -640,7 +641,7 @@ if print-edition { imprint( title: title, context-line: context-line, subtitle: subtitle, - generated: generated, site-url: site-url, copyright: copyright, + generated: generated, site-url: site-url, copyright: copyright, authors: authors, ) pagebreak() } diff --git a/tools/pdf/template.typ b/tools/pdf/template.typ index d4ebd3a93..af667b6d2 100644 --- a/tools/pdf/template.typ +++ b/tools/pdf/template.typ @@ -20,6 +20,7 @@ $endif$ version: sys.inputs.at("version"), site-url: sys.inputs.at("site-url"), copyright: sys.inputs.at("copyright"), + authors: sys.inputs.at("authors", default: ""), ) $body$ diff --git a/tools/pdf_authors.py b/tools/pdf_authors.py index 0e72826b3..176fd9e23 100644 --- a/tools/pdf_authors.py +++ b/tools/pdf_authors.py @@ -1,47 +1,267 @@ """Write the author list the imprint of the print edition prints. The authors are the GitHub accounts that contributed to the documentation -repository, most commits first. Anonymous contributions - commits whose e-mail -maps to no GitHub account - bot accounts and software agents are not listed. +repository, most commits first, each with the name its GitHub profile shows. +Anonymous contributions - commits whose e-mail maps to no GitHub account - bot +accounts, software agents and the IDs excluded in tools/pdf/print.yml are not +listed, and the names of excluded IDs are never looked up. The list is committed as tools/pdf/authors.yml, so a PDF build stays offline -and reproducible, and a changed list shows up in review. Invoked by -`task pdf:authors`; GITHUB_TOKEN or GH_TOKEN is used when set. +and reproducible, and a changed list shows up in review. The hand-maintained +`authors` section of tools/pdf/print.yml names authors whose profile shows no +name or who print with a title, and excludes IDs; the build applies it to the +committed list (`load_imprint_names`). Each run adds the authors that section +does not list yet, without a name, so every name to maintain is in one place +(`prefill_names`). Invoked by `task pdf:authors`. Requests use GITHUB_TOKEN or +GH_TOKEN, else the token of a logged-in GitHub CLI: without a token GitHub +allows 60 requests an hour, and a run takes one per author. """ from __future__ import annotations import json import os import re +import shutil +import subprocess +import sys +import time +import urllib.error +import urllib.parse import urllib.request +from dataclasses import dataclass, field from pathlib import Path +from typing import Callable import click import yaml REPOSITORY = "eccenca/documentation.eccenca.com" AUTHORS_YML = Path("tools/pdf/authors.yml") +# The hand-maintained names and exclusions: `authors` in the print configuration. +PRINT_YML = Path("tools/pdf/print.yml") API = "https://api.github.com" # Agents commit under ordinary accounts of type User, so they are named here. AGENT = re.compile(r"claude|codex", re.IGNORECASE) -def select_authors(contributors: list[dict]) -> list[dict]: +@dataclass(frozen=True) +class AuthorRules: + """The hand-maintained part of the author list, keyed by GitHub ID in lower case. + + `names` holds the name an author prints with - one the GitHub profile does + not show, or one with a title. `exclude` holds the IDs that are never listed. + """ + + names: dict[str, str] = field(default_factory=dict) + exclude: frozenset[str] = frozenset() + + def excludes(self, login: str) -> bool: + return login.casefold() in self.exclude + + def name(self, login: str) -> str | None: + return self.names.get(login.casefold()) + + +def load_author_rules(path: Path = PRINT_YML) -> AuthorRules: + """The `authors` section of the print configuration; without one, nothing is renamed or excluded.""" + data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + section = data.get("authors") or {} + if not isinstance(section, dict): + raise click.ClickException(f"{path}: authors holds `names` and `exclude`") + names = section.get("names") or {} + exclude = section.get("exclude") or [] + # A name left empty is not given: the imprint prints the profile's. + if not isinstance(names, dict) or not all( + isinstance(login, str) and (name is None or isinstance(name, str)) for login, name in names.items() + ): + raise click.ClickException(f"{path}: authors.names maps GitHub IDs to the names they print with") + if not isinstance(exclude, list) or not all(isinstance(login, str) for login in exclude): + raise click.ClickException(f"{path}: authors.exclude is a list of GitHub IDs") + return AuthorRules( + names={login.casefold(): name.strip() for login, name in names.items() if name and name.strip()}, + exclude=frozenset(login.casefold() for login in exclude), + ) + + +def select_authors(contributors: list[dict], rules: AuthorRules = AuthorRules()) -> list[dict]: """The contributors the imprint lists, in its order: most commits first, ties by ID, case-insensitive.""" people = [ c for c in contributors - if c.get("type") == "User" and c.get("login") and not AGENT.search(c["login"]) + if c.get("type") == "User" + and c.get("login") + and not AGENT.search(c["login"]) + and not rules.excludes(c["login"]) ] people.sort(key=lambda c: (-int(c["contributions"]), c["login"].casefold())) return [{"id": c["login"], "commits": int(c["contributions"])} for c in people] +def add_names(authors: list[dict], profile_name: Callable[[str], str | None]) -> list[dict]: + """Each author with the name its GitHub profile shows, or None when it shows none.""" + return [ + {"id": author["id"], "name": (profile_name(author["id"]) or "").strip() or None, "commits": author["commits"]} + for author in authors + ] + + +def imprint_names(authors: list[dict], rules: AuthorRules) -> tuple[list[str], list[str]]: + """The names the imprint prints, in order, and the IDs that print as themselves for want of a name. + + A name in the rules comes first, then the profile's. Excluded IDs are left + out, also when the committed list predates their exclusion. + """ + names: list[str] = [] + unnamed: list[str] = [] + for author in authors: + login = str(author["id"]) + if rules.excludes(login): + continue + name = rules.name(login) or author.get("name") + if not name: + unnamed.append(login) + names.append(name or login) + return names, unnamed + + +def load_imprint_names( + authors_yml: Path = AUTHORS_YML, print_yml: Path = PRINT_YML +) -> tuple[list[str], list[str]]: + """`imprint_names` for the committed author list and the print configuration.""" + data = yaml.safe_load(authors_yml.read_text(encoding="utf-8")) or {} + return imprint_names(data.get("authors") or [], load_author_rules(print_yml)) + + +def names_entry(login: str, profile_name: str | None) -> str: + """An `authors.names` line that gives no name, with the profile's name in its comment.""" + # Dumped, so a login YAML would read as something else - `null`, `yes` - is quoted. + key = yaml.safe_dump({login: None}, allow_unicode=True).rstrip("\n").removesuffix(" null") + note = f"GitHub profile: {' '.join(profile_name.split())}" if profile_name else "no name on the GitHub profile" + return f" {key} # {note}" + + +def prefill_names(text: str, authors: list[dict]) -> tuple[str, list[str]]: + """The print configuration with an `authors.names` entry for each author it neither lists nor excludes. + + Returns the text and the IDs added. An added entry gives no name, so the + imprint keeps printing the profile's until someone fills one in. The file is + edited as text, so its comments stay, and read back to make sure the new + entries are all that changed. + """ + before = yaml.safe_load(text) or {} + section = before.get("authors") or {} + listed = {str(login).casefold(): name for login, name in (section.get("names") or {}).items()} + known = set(listed) | {str(login).casefold() for login in (section.get("exclude") or [])} + missing = [author for author in authors if str(author["id"]).casefold() not in known] + if not missing: + return text, [] + ids = [str(author["id"]) for author in missing] + entries = [names_entry(login, author.get("name")) for login, author in zip(ids, missing)] + + lines = text.splitlines() + top = next((i for i, line in enumerate(lines) if re.match(r"authors:\s*(#.*)?$", line)), None) + if top is None: + lines += ["", "authors:", " names:", *entries, " exclude: []"] + else: + # The section runs until the next line that starts at the margin. + end = next( + (i for i in range(top + 1, len(lines)) if lines[i].strip() and not lines[i].startswith(" ")), + len(lines), + ) + names = next((i for i in range(top + 1, end) if re.match(r" names:", lines[i])), None) + if names is None: + lines[top + 1:top + 1] = [" names:", *entries] + else: + lines[names] = re.sub(r":\s*\{\s*\}", ":", lines[names], count=1) + last = names + for i in range(names + 1, end): + if not lines[i].strip(): + continue + if not lines[i].startswith(" "): + break + last = i + lines[last + 1:last + 1] = entries + result = "\n".join(lines) + "\n" + + after = yaml.safe_load(result) or {} + after_section = after.get("authors") or {} + after_names = {str(login).casefold(): name for login, name in (after_section.get("names") or {}).items()} + others = lambda data: {key: value for key, value in data.items() if key != "authors"} + if ( + after_names != {**listed, **{login.casefold(): None for login in ids}} + or (after_section.get("exclude") or []) != (section.get("exclude") or []) + or others(after) != others(before) + ): + raise click.ClickException( + f"cannot add {', '.join(ids)} to authors.names of the print configuration - add them by hand" + ) + return result, ids + + def next_page(link_header: str) -> str | None: """The `rel="next"` URL of a GitHub Link header, if there is one.""" match = re.search(r'<([^>]+)>;\s*rel="next"', link_header) return match.group(1) if match else None +TOKEN_HINT = "set GITHUB_TOKEN or GH_TOKEN, or log in with `gh auth login`" + + +def github_token() -> tuple[str | None, str | None]: + """The token for the GitHub API, and where it comes from. + + GITHUB_TOKEN or GH_TOKEN when set, else the token the GitHub CLI is logged in + with. Without either, requests go out unauthenticated. + """ + for variable in ("GITHUB_TOKEN", "GH_TOKEN"): + if os.environ.get(variable): + return os.environ[variable], variable + gh = shutil.which("gh") + if gh is None: + return None, None + try: + result = subprocess.run([gh, "auth", "token"], capture_output=True, text=True, timeout=10, check=False) + except (OSError, subprocess.SubprocessError): + return None, None + token = result.stdout.strip() if result.returncode == 0 else "" + return (token, "the GitHub CLI") if token else (None, None) + + +def api_failure(error: urllib.error.HTTPError, url: str, authenticated: bool) -> str: + """What a failed GitHub API request means, and what to do about it.""" + headers = error.headers or {} + reset, retry = headers.get("X-RateLimit-Reset"), headers.get("Retry-After") + limited = headers.get("X-RateLimit-Remaining") == "0" or retry or "rate limit" in str(error.reason).lower() + if error.code in (403, 429) and limited: + if reset and str(reset).isdigit(): + when = f"; it resets at {time.strftime('%H:%M', time.localtime(int(reset)))}" + elif retry and str(retry).isdigit(): + when = f"; retry in {retry} seconds" + else: + when = "" + advice = "" if authenticated else f" - {TOKEN_HINT} for 5,000 requests an hour" + return f"the GitHub API rate limit is exceeded{when}{advice}" + if error.code == 401: + return "GitHub rejected the token (HTTP 401) - check GITHUB_TOKEN, GH_TOKEN or `gh auth status`" + return f"the GitHub API answered HTTP {error.code} {error.reason} for {url}" + + +def get_json(url: str, token: str | None) -> tuple[object, str | None]: + """A GitHub API response, and the URL of its next page. + + A failed request ends the command with what it means, not with a traceback. + """ + request = urllib.request.Request(url, headers={"Accept": "application/vnd.github+json"}) + if token: + request.add_header("Authorization", f"Bearer {token}") + try: + with urllib.request.urlopen(request, timeout=30) as response: + return json.load(response), next_page(response.headers.get("Link", "")) + except urllib.error.HTTPError as error: + raise click.ClickException(api_failure(error, url, token is not None)) from error + except (urllib.error.URLError, TimeoutError) as error: + raise click.ClickException(f"cannot reach the GitHub API ({getattr(error, 'reason', error)})") from error + + def fetch_contributors(repository: str, token: str | None) -> list[dict]: """Every contributor of a repository, following the API's pagination. @@ -50,15 +270,17 @@ def fetch_contributors(repository: str, token: str | None) -> list[dict]: url: str | None = f"{API}/repos/{repository}/contributors?per_page=100" contributors: list[dict] = [] while url: - request = urllib.request.Request(url, headers={"Accept": "application/vnd.github+json"}) - if token: - request.add_header("Authorization", f"Bearer {token}") - with urllib.request.urlopen(request, timeout=30) as response: - contributors.extend(json.load(response)) - url = next_page(response.headers.get("Link", "")) + page, url = get_json(url, token) + contributors.extend(page) return contributors +def fetch_profile_name(login: str, token: str | None) -> str | None: + """The name a GitHub account's public profile shows, if it shows one.""" + profile, _ = get_json(f"{API}/users/{urllib.parse.quote(login)}", token) + return profile.get("name") + + class IndentedDumper(yaml.SafeDumper): """Indents list items under their key, as the repository's yamllint rules require.""" @@ -71,7 +293,8 @@ def authors_yaml(repository: str, authors: list[dict]) -> str: header = ( "---\n" "# Generated by `dec-tool pdf-authors` from the GitHub contributors of\n" - f"# {repository}. Do not edit - run `task pdf:authors`.\n" + f"# {repository} and the names their profiles show.\n" + "# Do not edit - run `task pdf:authors`. Names and exclusions: tools/pdf/print.yml.\n" ) body = yaml.dump( {"repository": repository, "authors": authors}, @@ -95,10 +318,32 @@ def authors_yaml(repository: str, authors: list[dict]) -> str: show_default=True, ) def pdf_authors(repository: str, output_file: str) -> None: - """Write the imprint's author list from the repository's GitHub contributors.""" - token = os.environ.get("GITHUB_TOKEN") or os.environ.get("GH_TOKEN") - authors = select_authors(fetch_contributors(repository, token)) + """Write the imprint's author list from the repository's GitHub contributors and their profile names.""" + token, source = github_token() + if source: + # Flushed, so the line comes before an error message also when the output is piped. + print(f"Using the GitHub token from {source}", flush=True) + else: + print( + "WARNING: no GitHub token - GitHub allows 60 requests an hour without one, and this run takes one " + f"per author; {TOKEN_HINT}", + file=sys.stderr, + ) + rules = load_author_rules() + authors = select_authors(fetch_contributors(repository, token), rules) if not authors: raise click.ClickException(f"no contributors found for {repository}") + authors = add_names(authors, lambda login: fetch_profile_name(login, token)) Path(output_file).write_text(authors_yaml(repository, authors), encoding="utf-8") print(f"{len(authors)} authors written to {output_file}") + prefilled, added = prefill_names(PRINT_YML.read_text(encoding="utf-8"), authors) + if added: + PRINT_YML.write_text(prefilled, encoding="utf-8") + print(f"{len(added)} authors added to authors.names in {PRINT_YML}: {', '.join(added)}") + _, unnamed = imprint_names(authors, load_author_rules()) + if unnamed: + print( + f"WARNING: {len(unnamed)} authors show no name on their GitHub profile and have none in " + f"{PRINT_YML} (authors.names), so the imprint prints their ID: {', '.join(unnamed)}", + file=sys.stderr, + ) From 5923313265bbad01e70f5c4e84934e47462c1f43 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Tue, 15 Sep 2026 14:55:34 +0200 Subject: [PATCH 03/15] leave content out of the print edition MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The print edition had no way to leave out single pages or parts of a page, and omit left a title and a note behind. - print.yml keys may name a page; omit drops pages and subtrees without a trace - the class print-exclude replaces a part of a page with a note pointing to the page online; the site and screen PDF are unchanged - omit the cmem-client API and the OSINT tutorial, and the INSERT listing of the Snowflake tutorial: 966 to 666 pages - spec §10 and backlog P18, authoring note in repo-conventions.md --- .claude/docs-guidelines/repo-conventions.md | 18 +++ docs/build/snowflake-tutorial/index.md | 2 +- tasks/backlog.md | 59 +++++++- tasks/spec.md | 145 +++++++++++++++++++- tests/test_build_pdf.py | 4 +- tests/test_build_pdf_print.py | 129 ++++++++++++++--- tools/build_pdf.py | 142 +++++++++++++------ tools/pdf/print.yml | 22 +-- 8 files changed, 447 insertions(+), 74 deletions(-) diff --git a/.claude/docs-guidelines/repo-conventions.md b/.claude/docs-guidelines/repo-conventions.md index b142e23db..f23ca681b 100644 --- a/.claude/docs-guidelines/repo-conventions.md +++ b/.claude/docs-guidelines/repo-conventions.md @@ -204,6 +204,24 @@ Known drift to fix when touching a page: 32 `shell-session` blocks in hand-writt numbering restarts at 1 in the rendered page. Check the rendered result for any step list you touch. - `sane_lists` is enabled: an unordered item does not continue an ordered list. +## Leaving content out of the print edition + +The print edition of the PDF (`task pdf:print`, `tasks/spec.md` §10) can leave out content that only makes sense online. + +- A subtree or a single page: an `omit` entry under `sections` in `tools/pdf/print.yml`, keyed by its `docs/` path. + The page itself carries no marker, so this works for generated pages too. +- A part of a page: the class `print-exclude` on the block. The print edition prints a note in its place that points + to the page online; the site and the screen PDF show the part unchanged. + +| Block | Markup | +| --- | --- | +| code block | `sql { .print-exclude }` right after the backticks of the opening fence | +| admonition or collapsible block | `??? example print-exclude "INSERT query"` | +| paragraph | `{ .print-exclude }` on the line after the paragraph | +| several blocks | `
` … `
`, with blank lines around the content | + +A generated page cannot carry the class: the next generator run removes it. + ## Editorial decisions beyond the style guide `style-guide.md` is a transcription of the Confluence page and is not extended locally. diff --git a/docs/build/snowflake-tutorial/index.md b/docs/build/snowflake-tutorial/index.md index 3f898c32c..43d2d4c84 100644 --- a/docs/build/snowflake-tutorial/index.md +++ b/docs/build/snowflake-tutorial/index.md @@ -89,7 +89,7 @@ Here you can populate some test data with the following SQL query: ??? example "INSERT query" - ```sql + ```sql { .print-exclude } INSERT INTO product(product_id,product_name,height,width,depth,weigth,product_manager,price) VALUES ('I241-8776317','Strain Compensator',12,68,15,8,'Baldwin.Dirksen@company.org','0,50 EUR') ,('D215-3449390','Gauge Crystal',77,58,19,15,'Wanja.Hoffmann@company.org','2,00 EUR') diff --git a/tasks/backlog.md b/tasks/backlog.md index 33c3ed484..9223161ae 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -12,7 +12,8 @@ Every task names how it is verified. A task is not done until that verification Spec §4, D1-D11: BoD, A4, black and white on 80 g, no ISBN, authors by name (first GitHub IDs, revised 2026-09-15), separate screen and print editions, section modes with A.3 and Release Notes as lists, page references and URL footnotes, -no logo or version on text pages, 10 pt body, optional Ghostscript normalization. +no logo or version on text pages, 10 pt body, optional Ghostscript normalization. D12-D14, decided +2026-09-15: excluding content from the print edition (spec §10), implemented by P18. --- @@ -342,6 +343,61 @@ so it is not forgotten; it has its own spec. --- +## P18 - Content exclusion - **done** + +Spec §10, D12-D14: leave subtrees, pages and parts of a page out of the print edition. + +- `sections` keys in `tools/pdf/print.yml` may name a page (`.md`), which accepts `omit` only; a nested + key, or one that matches no page in `nav.yml`, fails the build +- `omit` drops a page or a subtree without a title or a note. For directories this revises P8, which + left the section's title and a note naming the online edition +- dropping a section's index page keeps its other pages under a heading with the navigation title +- in the print edition, elements with the class `print-exclude` are removed from the page's article + before ids, links and headings are processed. A note takes their place: "This print edition leaves + out a part of this page. The online edition has the full details:" and the page's online address, + with the anchor of the heading before the part; consecutive parts share one note. The build logs the + parts removed per page +- the site and the screen PDF ignore the class +- documentation: the `sections` comment in `print.yml`, and a line for authors in + `.claude/docs-guidelines/repo-conventions.md` on the class, which generated pages cannot carry +- configuration: `develop/cmem-client-api/: omit`, `build/tutorial-how-to-link-ids-to-osint/: omit` (all + 7 pages), and `{ .sql .print-exclude }` on the SQL code block of the Snowflake tutorial - lines 92-1094 + of `docs/build/snowflake-tutorial/index.md` on 2026-09-15, inside `??? example "INSERT query"`, which + keeps its title + +**Verify:** unit tests for page keys, the mode check, nesting, a dropped index page, the removal with +one note per run of parts and its address, and no removal in the screen edition. `task pdf:print` with +the three examples: G.5 and A.16 leave no heading or note, the `INSERT query` block of +the Snowflake tutorial holds the note instead of the listing, and the page count drops by about 300 +(spec §10). `task pdf`: page count and text unchanged apart +from the stamp. `task check` passes. +**Est:** small to medium. **Depends on:** P8, P9. + +**Done 2026-09-15:** + +- `SectionRule` knows page rules (`page`, `matches`). `load_section_rules` accepts page keys with `omit` + only and rejects nested keys. +- `omit_section` drops a page or a subtree without a trace. An index page omitted on its own leaves its + navigation title as a heading. +- `exclude_parts` replaces each run of `.print-exclude` elements with the note, whose address carries the + anchor of the heading before the part. `merge_pages` returns the parts left out per page, which the + build logs. List tables skip marked elements as well. +- `print.yml` omits `develop/cmem-client-api/` (75 pages) and `build/tutorial-how-to-link-ids-to-osint/` + (7 pages). The SQL block of the Snowflake tutorial carries `sql { .print-exclude }`. Authors find the + markup in `.claude/docs-guidelines/repo-conventions.md`. +- Measured: + - The print edition drops from 966 to 666 pages, the Snowflake tutorial from 46 to 22. + - No bookmark is left for G.5 or A.16. + - The note stands inside the `INSERT query` block (p. 120), with the anchor + `#1-create-a-database-in-snowflake`. + - No page runs into the footer and no line runs past the text column. + - The screen PDF keeps 1683 pages, its text unchanged and the listing in it. + - 134 unit tests pass. + - The `task check` steps pass; yamllint passes on the tracked files, while the untracked `scratch/` + folder still fails it. + +--- + ## Sequencing ```text @@ -354,6 +410,7 @@ P1 ──> P2 ──> P3 ──> P4 ──> P5 ──> P7 └──────────────┼──> P13 <────┘ └──> P14 ──> P16 P17 after P8 and P14 +P18 after P8 and P9 ``` P1, P6, P8, P10, P11 and P12 can start now. diff --git a/tasks/spec.md b/tasks/spec.md index 9d5e3a5d8..48ab3266b 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -2,7 +2,8 @@ **Status:** accepted 2026-09-14 - all decisions made (§4). Backlog P0-P12 implemented 2026-09-15 (`task pdf:print`, 962 pages); P13-P17 open. The backlog's "Done" notes record where the implementation -refines this spec. +refines this spec. §10, excluding content from the print edition, was decided on 2026-09-15 (D12-D14) +and implemented the same day (backlog P18): the print edition has 666 pages. **Branch:** `feature/print-on-demand`, based on `main` at `c20d74b94` (PDF export merged). **Goal:** a *book block* - the interior file of a printed, perfect-bound book - built next to the screen PDF, which BoD accepts without rework. @@ -157,7 +158,7 @@ are left out of the book block - Ghostscript refuses PDF/X output while a page c ## 4. Decisions -All made 2026-09-14. +D1-D11 made 2026-09-14; D12-D14 made 2026-09-15 (§10). | # | Question | Decision | | :-- | :-- | :-- | @@ -172,6 +173,9 @@ All made 2026-09-14. | D9 | Logo and version in the running header | title page and imprint only (R1) | | D10 | Body type size | keep 10 pt and tighten the spacing (§5). A smaller body, 9 pt or even 8 pt, only if the page limit is still exceeded - not needed for the default configuration | | D11 | Normalization | an **optional** Ghostscript pass after Typst: PDF/X-4, CMYK, all images at 300 dpi (§7) | +| D12 | Subtrees and pages | path rules in `tools/pdf/print.yml`: a `sections` key may name a page, which accepts `omit` only; no front matter property, no `.pdfexclude` (§10) | +| D13 | Parts of a page | the class `print-exclude`, effective in the print edition only; the site and the screen PDF are unchanged (§10) | +| D14 | What stands in for excluded content | pages and subtrees: nothing, also for a directory's `omit` (revises §5); parts of a page: one note per run of parts, pointing to the page in the online edition for the full details (§10) | ## 5. Page budget @@ -199,8 +203,9 @@ measure its page cost before deciding. ### Section modes -`full` prints the section as today. `omit` leaves it out; the part contents point to the online -edition. `list` reduces it to a two-column table: +`full` prints the section as today. `omit` leaves it out without a trace in the text (D14, revised +2026-09-15); the imprint names the online edition as the complete reference. `list` reduces it to a +two-column table: | Section | `list` renders | Measured | | :-- | :-- | :-- | @@ -369,6 +374,9 @@ the same task, for checking the palette (P11). - every image at 300 ppi or more at its printed size, or listed and accepted - with normalization: the file declares PDF/X-4, carries a FOGRA39 output intent, CMYK images only, no image above 300 ppi +- a page key in `print.yml` drops that page; `omit` leaves no title and no note for a page or a section +- in the print edition, each run of `.print-exclude` parts is replaced by one note with the page's + online address; the site and the screen PDF show the parts unchanged - `task check` and `task test:unit` pass ## 9. Out of scope @@ -377,3 +385,132 @@ the same task, for checking the palette (P11). - ISBN and retail distribution (D3, maybe later) - EPUB or other e-book formats - translations + +## 10. Excluding content from the print edition + +**Status:** decided and implemented 2026-09-15 (D12-D14 in §4, backlog P18). With the three examples +configured, the print edition has 666 pages instead of 966. + +The section modes (§5) shorten whole reference sections. Some content is unfit for paper at a finer +grain: + +| Grain | Example | Print pages | +| :-- | :-- | --: | +| subtree | G.5 cmem-client: Python API, `develop/cmem-client-api/` - 75 pages, 74 of them generated | 207 | +| subtree | A.16 How to link IDS to OSINT, `build/tutorial-how-to-link-ids-to-osint/` - all 7 pages (decided 2026-09-15) | 68 | +| page | none configured yet; a page key leaves out a single page when one needs it | - | +| part of a page | A.14 Connect to Snowflake: the SQL code block of the collapsed `??? example "INSERT query"` block, lines 92-1094 of `docs/build/snowflake-tutorial/index.md` on 2026-09-15 - about 1,000 lines (decided 2026-09-15) | 25 (pp. 122-146) | + +Measured in the print edition of 2026-09-15 (966 pages). Excluding all three saves about 300 pages. + +### Constraints + +- The build reads the rendered site, not the Markdown. A marker for a part of a page must survive + Zensical's rendering; a rule for a page or subtree must be decidable from the paths in `nav.yml`. +- Generated pages are rewritten wholesale: `task update:cmemc` and `task update:cmem-client-api` run + `rm -rf /*`, and `dec-tool update-di-reference` deletes its whole tree. Whatever is stored in a + generated page, or next to it, is lost on the next run. +- The site and the screen PDF stay as they are (D6). + +### Options + +| Option | Subtree | Page | Part | Verdict | +| :-- | :-- | :-- | :-- | :-- | +| front matter property, e.g. `print: exclude` | each page marked | yes | no | rejected: lost on generated pages; a subtree means marking every page; no single place shows what the book leaves out | +| `.pdfexclude` file in gitignore syntax | yes | yes | no | rejected: a second configuration next to `print.yml`, with a new format and a discovery rule, spread over the tree; deleted with a generated directory | +| path rules in `tools/pdf/print.yml` | yes - `omit` exists | yes, once a key may name a page | no | **chosen** for subtrees and pages (D12) | +| comment pair `` … `` | no | no | yes | works, not proposed: the pair must stay siblings, a single block costs two extra lines, and a misspelt marker is ignored without a trace | +| class `print-exclude` on the rendered element | no | no | yes | **chosen** for parts of a page (D13) | +| CSS selectors per page in `print.yml` | no | no | yes | rejected: ties the configuration to theme markup | +| `exclude_docs` or `not_in_nav` in `mkdocs.yml` | yes | yes | no | rejected: removes the pages from the site as well | + +Verified on 2026-09-15 in a scratch project with Zensical 0.0.62 and the Markdown extensions of +`mkdocs.yml`: + +- **Comment pairs:** they arrive in the HTML as unescaped comments. Both comments of a pair stay + side by side, at top level and inside an admonition, a content tab and a list item. +- **The class:** it lands on the element in every spelling listed below. Removing the `.print-exclude` + elements leaves exactly the unmarked content. +- **Front matter:** an unknown front matter key builds without a warning. + +### Design + +#### Subtrees and pages: path rules in `print.yml` (D12) + +A key under `sections` names a docs directory, ending in `/` as today, or a single page, ending in +`.md`: + +```yaml +sections: + develop/cmem-client-api/: omit + build/tutorial-how-to-link-ids-to-osint/: omit + # a single page: /index.md: omit +``` + +- A directory key keeps its three modes (§5). `omit` drops the section without a title or a note (D14). + The first implementation (P8) left both; the imprint names the online edition as the complete + reference instead. +- A page key accepts `omit` only; any other mode fails the build. The page is dropped without a note. +- Dropping the index page of a section keeps its other pages, under a heading with the section's + navigation title - the shape Release Notes already has (§3). +- Keys do not nest: a key inside a directory that another key shortens fails the build. +- A key that matches no page in `nav.yml` fails the build, as today. +- A link to a dropped page prints the page's online address in a footnote, like any link that leaves + the book (R4). Within a section that `list` or `omit` shortens, it prints as text, as today. + +#### Parts of a page: the class `print-exclude` (D13) + +One class, spelled the way the element takes it: + +````markdown +??? example print-exclude "INSERT query" + + ```sql + INSERT INTO product(...) VALUES ... + ``` + +```{ .sql .print-exclude } +SELECT ... +``` + +A paragraph the print edition leaves out. +{ .print-exclude } + +
+ +Several blocks - also inside a list item, a content tab or an admonition. + +
+```` + +- The print edition removes every element with the class from a page's article, before ids, links and + headings are processed. The class has no effect on the site and the screen PDF. +- A heading inside a removed part leaves the numbering and the contents. Links to it print as text, the + rule for any link without a target. +- The build logs how many parts it removed per page, so a marker that no longer matches shows. +- A note takes the place of a removed part (D14): *This print edition leaves out a part of this page. + The online edition has the full details:* followed by the page's online address, with the anchor of + the heading the part belongs to. Consecutive removed parts share one note. +- In the Snowflake tutorial the class goes on the SQL code block, as `{ .sql .print-exclude }` on its + opening fence: exactly lines 92-1094 are removed. The `??? example "INSERT query"` block around it keeps + its title and holds the note. +- A generated page cannot carry the class: its generator has to emit it, or a path rule drops the page. +- The name follows the edition. The site does not style the class; a later `@media print` rule could + use it for printing from the browser. + +### Implementation outline + +- **Rules:** `load_section_rules` accepts `.md` keys, rejects any mode but `omit` for them and rejects + nested keys. +- **Page `omit`:** `apply_section_rules` drops the page's entry. For an index page it inserts a heading + entry with the navigation title, which `NavEntry.title` already carries. +- **Parts:** in the print edition, `page_article` replaces the `.print-exclude` elements with the note - + one per run of consecutive parts - and counts them. +- **`omit` for directories:** no title and no note any more. `test_omit_drops_the_pages_and_leaves_a_heading_with_a_note` + and `test_merge_renders_an_omitted_section_as_its_title_and_a_note` change with it. +- **Tests:** `tests/test_build_pdf_print.py` gets page keys, the mode check, nesting, a dropped index + page, and the removal with its note and online address in the print edition only. +- **Documentation:** the `sections` comment in `print.yml`, and for authors a line in + `.claude/docs-guidelines/repo-conventions.md` on the class, including that generated pages cannot + carry it. +- **Effort:** small to medium; backlog P18. diff --git a/tests/test_build_pdf.py b/tests/test_build_pdf.py index b5229759f..38eb1eb6d 100644 --- a/tests/test_build_pdf.py +++ b/tests/test_build_pdf.py @@ -74,7 +74,7 @@ def test_merge_breaks_chapters_adds_section_headings_and_drops_web_chrome(tmp_pa NavEntry(depth=1, md="release-notes/v1/index.md"), NavEntry(depth=1, md="release-notes/v2/index.md"), ] - doc, missing = merge_pages(entries, site, "https://example.org/latest/") + doc, missing, _ = merge_pages(entries, site, "https://example.org/latest/") assert missing == ["release-notes/v2/index.md"] assert str(doc.body) == ( '

Release Notes

' @@ -92,7 +92,7 @@ def test_part_cover_is_the_title_then_the_diagram_the_page_shows_above_it(tmp_pa '

Build

Intro

' '' ) - doc, _ = merge_pages([NavEntry(depth=0, md="build/index.md")], site, "https://example.org/latest/") + doc, _, _ = merge_pages([NavEntry(depth=0, md="build/index.md")], site, "https://example.org/latest/") assert str(doc.body) == ( '
' '

Build

You are here

' diff --git a/tests/test_build_pdf_print.py b/tests/test_build_pdf_print.py index 018b1031c..554957286 100644 --- a/tests/test_build_pdf_print.py +++ b/tests/test_build_pdf_print.py @@ -1,4 +1,4 @@ -"""Test the print edition's section modes: full, list and omit""" +"""Test the print edition's section modes - full, list and omit - and its excluded parts of a page""" import click import pytest @@ -76,6 +76,41 @@ def test_an_unknown_mode_fails(tmp_path): load_section_rules(path) +def test_a_key_may_name_a_single_page(tmp_path): + path = tmp_path / "print.yml" + path.write_text("sections:\n develop/cmem-client-api/: omit\n build/tutorial/step/index.md: omit\n") + subtree, page = load_section_rules(path) + assert (subtree, page) == ( + SectionRule("develop/cmem-client-api/", "omit"), + SectionRule("build/tutorial/step/index.md", "omit"), + ) + assert subtree.matches("develop/cmem-client-api/api-reference/client/index.md") + assert page.matches("build/tutorial/step/index.md") + assert not page.matches("build/tutorial/step/index.md/more") + + +@pytest.mark.parametrize("mode", ["list", "full"]) +def test_a_page_key_accepts_only_omit(tmp_path, mode): + path = tmp_path / "print.yml" + path.write_text(f"sections:\n build/tutorial/step/index.md: {mode}\n") + with pytest.raises(click.ClickException, match="only `omit`"): + load_section_rules(path) + + +@pytest.mark.parametrize( + "keys", + [ + " build/: list\n build/reference/: omit\n", + " build/tutorial/: omit\n build/tutorial/step/index.md: omit\n", + ], +) +def test_keys_do_not_nest(tmp_path, keys): + path = tmp_path / "print.yml" + path.write_text("sections:\n" + keys) + with pytest.raises(click.ClickException, match="nest"): + load_section_rules(path) + + def test_list_keeps_the_overview_pages_and_drops_what_they_list(): entries, dropped = apply_section_rules(REFERENCE, [SectionRule("build/reference/", "list")]) assert entries == [ @@ -125,16 +160,35 @@ def test_list_turns_the_titles_an_overview_empties_into_one_table(): assert dropped == {CATEGORIES[3].md, CATEGORIES[4].md, CATEGORIES[6].md} -def test_omit_drops_the_pages_and_leaves_a_heading_with_a_note(): +def test_omit_drops_a_subtree_without_a_trace(): entries, dropped = apply_section_rules(REFERENCE, [SectionRule("build/reference/", "omit")]) - assert entries == [ - NavEntry(0, "build/index.md"), - NavEntry(1, generated=Generated("note", "build/reference/", "omit", ["build/reference/index.md"])), - NavEntry(1, "build/spark.md"), - ] + assert entries == [NavEntry(0, "build/index.md"), NavEntry(1, "build/spark.md")] assert len(dropped) == 6 +def test_omit_drops_a_single_page(): + entries, dropped = apply_section_rules(REFERENCE, [SectionRule("build/spark.md", "omit")]) + assert entries == REFERENCE[:-1] + assert dropped == {"build/spark.md"} + + +def test_omitting_an_index_page_keeps_its_pages_under_the_section_title(): + nav = [ + NavEntry(0, "build/index.md", "Build"), + NavEntry(1, "build/tutorial/index.md", "Tutorial"), + NavEntry(2, "build/tutorial/step/index.md", "Step"), + NavEntry(1, "build/spark.md", "Spark"), + ] + entries, dropped = apply_section_rules(nav, [SectionRule("build/tutorial/index.md", "omit")]) + assert entries == [nav[0], NavEntry(1, title="Tutorial"), nav[2], nav[3]] + assert dropped == {"build/tutorial/index.md"} + + +def test_a_page_key_that_names_no_page_of_the_navigation_fails(): + with pytest.raises(click.ClickException, match="build/missing.md"): + apply_section_rules(REFERENCE, [SectionRule("build/missing.md", "omit")]) + + def test_omitting_a_whole_part_without_a_page_leaves_nothing_of_it(): entries, dropped = apply_section_rules(RELEASES, [SectionRule("release-notes/", "omit")]) assert entries == [NavEntry(0, "tutorials/index.md")] @@ -186,7 +240,7 @@ def test_merge_renders_notes_and_tables_and_unlinks_what_the_section_drops(tmp_p ] rules = [SectionRule("build/reference/", "list"), SectionRule("release-notes/", "list", ("Release", "Summary"))] dropped = {"build/reference/aggregator/average.md", first, second} - doc, missing = merge_pages(entries, site, "https://example.org/26.2/", rules, dropped) + doc, missing, _ = merge_pages(entries, site, "https://example.org/26.2/", rules, dropped) assert missing == [] html = str(doc.body) text = " ".join(doc.body.get_text(" ").split()) @@ -222,7 +276,7 @@ def test_merge_renders_notes_and_tables_and_unlinks_what_the_section_drops(tmp_p def test_merge_renders_the_emptied_titles_as_one_table_of_their_pages(tmp_path): site = tmp_path / "site" write_page(site, "build/reference/transformer/Date/duration.md", '

Duration

') - doc, _ = merge_pages([NavEntry(3, generated=CATEGORY_TABLE)], site, "https://example.org/26.2/") + doc, _, _ = merge_pages([NavEntry(3, generated=CATEGORY_TABLE)], site, "https://example.org/26.2/") table = doc.find("table", class_="print-list") rows = [[cell.get_text(" ", strip=True) for cell in row.find_all(["th", "td"])] for row in table.find_all("tr")] # A page without a navigation title is named by its own title. @@ -236,13 +290,52 @@ def test_merge_renders_the_emptied_titles_as_one_table_of_their_pages(tmp_path): assert {cell.get("style") for cell in table.find_all(["th", "td"])} == {"text-align: left;"} -def test_merge_renders_an_omitted_section_as_its_title_and_a_note(tmp_path): +PART_NOTE = "This print edition leaves out a part of this page. The online edition has the full details:" + + +def test_the_print_edition_replaces_each_run_of_excluded_parts_with_one_note(tmp_path): site = tmp_path / "site" - write_page(site, "build/reference/index.md", '

Task and Operator Reference

') - entries = [NavEntry(1, generated=Generated("note", "build/reference/", "omit", ["build/reference/index.md"]))] - doc, _ = merge_pages(entries, site, "https://example.org/26.2/", [SectionRule("build/reference/", "omit")], set()) - assert doc.body.find("h2").get_text() == "Task and Operator Reference" - assert ( - "The Task and Operator Reference is not part of this print edition. It is part of the online edition: " - "https://example.org/26.2/build/reference/" - ) in " ".join(doc.body.get_text(" ").split()) + write_page( + site, "build/snowflake/index.md", + '

Connect

Intro.

' + '

Data

Populate:

' + '
INSERT query' + '
INSERT INTO product
\n' + '

Second part.

' + '

Kept between.

' + '

Third part.

', + ) + doc, missing, excluded = merge_pages( + [NavEntry(0, "build/snowflake/index.md")], site, "https://example.org/26.2/", print_edition=True + ) + text = " ".join(doc.body.get_text(" ").split()) + # Two runs, each after the heading "Data", whose anchor the online address carries. + assert text.count(f"{PART_NOTE} https://example.org/26.2/build/snowflake/#data") == 2 + for gone in ("INSERT", "Second part.", "Third part."): + assert gone not in text + for kept in ("Intro.", "Populate:", "Kept between."): + assert kept in text + # A marked element inside a marked one goes with it and is not counted again. + assert excluded == {"build/snowflake/index.md": 3} + assert missing == [] + + +def test_the_screen_edition_keeps_the_excluded_parts(tmp_path): + site = tmp_path / "site" + write_page(site, "build/snowflake/index.md", '

Connect

On screen.

') + doc, _, excluded = merge_pages([NavEntry(0, "build/snowflake/index.md")], site, "https://example.org/26.2/") + text = doc.body.get_text(" ") + assert "On screen." in text and "leaves out" not in text + assert excluded == {} + + +def test_a_list_table_summarizes_a_page_by_its_first_paragraph_that_prints(tmp_path): + site = tmp_path / "site" + write_page( + site, "release-notes/2026/cm-26-2/index.md", + '

Corporate Memory 26.2.1

Online only.

The second release.

', + ) + table = Generated("table", "release-notes/", "list", ["release-notes/2026/cm-26-2/index.md"], ("Release", "Summary")) + doc, _, _ = merge_pages([NavEntry(2, generated=table)], site, "https://example.org/26.2/", print_edition=True) + rows = [[cell.get_text(" ", strip=True) for cell in row.find_all(["th", "td"])] for row in doc.find_all("tr")] + assert rows == [["Release", "Summary"], ["Corporate Memory 26.2.1", "The second release."]] diff --git a/tools/build_pdf.py b/tools/build_pdf.py index 371924e4c..88fd6d898 100644 --- a/tools/build_pdf.py +++ b/tools/build_pdf.py @@ -77,6 +77,11 @@ # A `groups` table: the navigation titles under an overview page - the operator # categories of the reference - and their pages, under the overview's title. GROUP_COLUMNS = ("Category", "Pages") +# The class that leaves a part of a page out of the print edition, and the note +# that takes its place (tasks/spec.md, §10). +PRINT_EXCLUDE = "print-exclude" +EXCLUDED_NOTE = "print-excluded" +PART_NOTE = "This print edition leaves out a part of this page. The online edition has the full details:" # The print edition's text column is 16 cm wide, and BoD asks for images at # 300 dpi at their printed size (tasks/spec.md, §2). TEXT_WIDTH_PT = 16 / 2.54 * 72 @@ -87,10 +92,10 @@ class Generated: """Content the print edition sets in place of a section's pages. - A `note` names the online edition for a section in `list` or `omit` mode; a - `table` lists pages that no overview page of the section lists; `groups` - lists the navigation titles whose pages an overview page lists, each with - the names of its pages. + A `note` names the online edition for a section in `list` mode; a `table` + lists pages that no overview page of the section lists; `groups` lists the + navigation titles whose pages an overview page lists, each with the names + of its pages. """ kind: str @@ -119,12 +124,20 @@ class NavEntry: @dataclass class SectionRule: - """How the print edition prints the pages under one docs/ directory.""" + """How the print edition prints the pages under one docs/ directory, or a single page.""" prefix: str mode: str columns: tuple[str, str] = DEFAULT_COLUMNS + @property + def page(self) -> bool: + """Whether the rule names a single page rather than a directory.""" + return self.prefix.endswith(".md") + + def matches(self, md: str) -> bool: + return md == self.prefix if self.page else md.startswith(self.prefix) + def load_site_config() -> dict: """Read mkdocs.yml, tolerating the `!!python/name:` handles it carries. @@ -212,28 +225,38 @@ def load_nav_entries(nav_yml: Path = NAV_YML) -> list[NavEntry]: # section's overview pages and drops the pages they list; pages no overview # lists become a table of title and first paragraph, and navigation titles left # without pages become one table of titles and page names. `omit` drops a -# section and leaves a note naming the online edition. +# section or a single page without a trace (tasks/spec.md, §10). def load_section_rules(path: Path = PRINT_YML) -> list[SectionRule]: """The section modes of the print edition. - A mode is given either as the value of a section, or as `mode` in a mapping - that may also name the two `columns` of the section's tables. + A key names a docs/ directory, or a single page by its `.md` path, which + accepts only `omit`. A mode is given either as the value of a key, or as + `mode` in a mapping that may also name the two `columns` of the section's + tables. Keys do not nest. """ data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} rules = [] - for prefix, value in (data.get("sections") or {}).items(): + for key, value in (data.get("sections") or {}).items(): spec = value if isinstance(value, dict) else {"mode": value} mode = spec.get("mode", "full") if mode not in SECTION_MODES: raise click.ClickException( - f"{path}: section {prefix} has mode {mode!r}, not one of {', '.join(SECTION_MODES)}" + f"{path}: section {key} has mode {mode!r}, not one of {', '.join(SECTION_MODES)}" ) columns = tuple(spec.get("columns") or DEFAULT_COLUMNS) if len(columns) != 2: - raise click.ClickException(f"{path}: section {prefix} needs two columns, not {len(columns)}") - rules.append(SectionRule(str(prefix).rstrip("/") + "/", mode, columns)) + raise click.ClickException(f"{path}: section {key} needs two columns, not {len(columns)}") + name = str(key) + rule = SectionRule(name if name.endswith(".md") else name.rstrip("/") + "/", mode, columns) + if rule.page and mode != "omit": + raise click.ClickException(f"{path}: {rule.prefix} names a page, which accepts only `omit`, not {mode!r}") + rules.append(rule) + for rule in rules: + outer = next((o for o in rules if o is not rule and not o.page and rule.prefix.startswith(o.prefix)), None) + if outer is not None: + raise click.ClickException(f"{path}: {rule.prefix} lies inside {outer.prefix} - section keys do not nest") return rules @@ -360,17 +383,22 @@ def drop_empty_headings(entries: list[NavEntry]) -> list[NavEntry]: def omit_section(entries: list[NavEntry], rule: SectionRule) -> tuple[list[NavEntry], set[str]]: - """Drop a section. A section with a page of its own leaves its title and a note in its place.""" - root = rule.prefix + "index.md" + """Drop a subtree or a single page, without a trace in the text (tasks/spec.md, §10). + + A dropped page with pages beneath it that stay - an index page omitted on its + own - leaves the section's navigation title as a heading, so those pages keep + their place. + """ result: list[NavEntry] = [] dropped: set[str] = set() - for entry in entries: - if entry.md and entry.md.startswith(rule.prefix): - dropped.add(entry.md) - if entry.md == root: - result.append(NavEntry(entry.depth, generated=Generated("note", rule.prefix, "omit", [root]))) + for i, entry in enumerate(entries): + if not (entry.md and rule.matches(entry.md)): + result.append(entry) continue - result.append(entry) + dropped.add(entry.md) + following = entries[i + 1] if i + 1 < len(entries) else None + if following is not None and following.depth > entry.depth and not (following.md and rule.matches(following.md)): + result.append(NavEntry(entry.depth, title=entry.title or section_name(entry.md.rsplit("/", 1)[0]))) return drop_empty_headings(result), dropped @@ -378,8 +406,9 @@ def apply_section_rules(entries: list[NavEntry], rules: list[SectionRule]) -> tu """Shorten the navigation by the section modes. Returns the entries and the pages dropped from them.""" dropped: set[str] = set() for rule in rules: - if not any(e.md and e.md.startswith(rule.prefix) for e in entries): - raise click.ClickException(f"{PRINT_YML}: no page in {NAV_YML} lies under {rule.prefix}") + if not any(e.md and rule.matches(e.md) for e in entries): + where = f"{rule.prefix} is not a page" if rule.page else f"no page lies under {rule.prefix}" + raise click.ClickException(f"{PRINT_YML}: {where} in {NAV_YML}") if rule.mode == "list": entries, gone = list_section(entries, rule) elif rule.mode == "omit": @@ -580,6 +609,10 @@ def summary_table(doc: BeautifulSoup, generated: Generated, site_dir: Path): table, body = list_table(doc, generated.columns) for md in generated.pages: article = page_article(site_dir, md) + # A part the print edition leaves out does not summarize the page either. + for part in article.select(f".{PRINT_EXCLUDE}") if article is not None else (): + if not part.decomposed: + part.decompose() heading = article.find("h1") if article is not None else None row = doc.new_tag("tr") title = list_cell(doc) @@ -636,20 +669,38 @@ def render_generated(doc: BeautifulSoup, entry: NavEntry, site_dir: Path, public return [groups_table(doc, generated, site_dir)] name = generated.name or page_title(site_dir, generated.pages[0]) url = section_url(generated, site_dir, public_base) - elements = [] - if generated.mode == "omit": - heading = doc.new_tag(f"h{min(6, entry.depth + 1)}") - heading.string = name - elements.append(heading) - text = f"The {name} is not part of this print edition. It is part of the online edition: {url}" - else: - text = f"This print edition lists the {name} in short. The complete section is part of the online edition: {url}" note = doc.new_tag("div", attrs={"class": "admonition info"}) paragraph = doc.new_tag("p") - paragraph.string = text + paragraph.string = ( + f"This print edition lists the {name} in short. The complete section is part of the online edition: {url}" + ) note.append(paragraph) - elements.append(note) - return elements + return [note] + + +def exclude_parts(doc: BeautifulSoup, article, page_url: str) -> int: + """Replace the parts of a page marked `print-exclude` with a note naming the page online (tasks/spec.md, §10). + + The note's address carries the anchor of the heading before the part, and + consecutive parts share one note. A marked element inside another marked one + goes with it. Returns the number of parts removed. + """ + parts = [part for part in article.select(f".{PRINT_EXCLUDE}") if part.find_parent(class_=PRINT_EXCLUDE) is None] + for part in parts: + previous = next( + (node for node in part.previous_siblings if not (isinstance(node, NavigableString) and not node.strip())), + None, + ) + if previous is not None and not isinstance(previous, NavigableString) and EXCLUDED_NOTE in previous.get("class", []): + part.decompose() + continue + heading = part.find_previous(HEADING, id=True) + note = doc.new_tag("div", attrs={"class": ["admonition", "info", EXCLUDED_NOTE]}) + paragraph = doc.new_tag("p") + paragraph.string = f"{PART_NOTE} {page_url}" + (f"#{heading['id']}" if heading is not None else "") + note.append(paragraph) + part.replace_with(note) + return len(parts) def merge_pages( @@ -658,7 +709,8 @@ def merge_pages( public_base: str, rules: list[SectionRule] | None = None, dropped: set[str] | None = None, -) -> tuple[BeautifulSoup, list[str]]: + print_edition: bool = False, +) -> tuple[BeautifulSoup, list[str], dict[str, int]]: """Merge the built articles along the navigation into one HTML document. Every top-level entry is a part. It is preceded by a chapter break, so a @@ -666,13 +718,16 @@ def merge_pages( `part_cover`); a part without a page gets its title and the contents marker. Generated entries of the print edition's section modes are rendered where they stand, and a shortened section's links to the pages it drops print as - text. Returns the document and the navigation pages that had no built HTML. + text. In the print edition, the parts of a page marked `print-exclude` give + way to a note (`exclude_parts`). Returns the document, the navigation pages + that had no built HTML, and the number of parts left out per page. """ doc = BeautifulSoup('', "html.parser") valid_ids = {url_to_section_id(md_to_url_path(e.md)) for e in entries if e.md} dropped_ids = {url_to_section_id(md_to_url_path(md)) for md in dropped or ()} - shortened = [rule.prefix for rule in rules or () if rule.mode != "full"] + shortening = [rule for rule in rules or () if rule.mode != "full"] missing: list[str] = [] + excluded: dict[str, int] = {} for entry in entries: if entry.generated is not None: @@ -695,20 +750,25 @@ def merge_pages( if article is None: missing.append(entry.md) continue + if print_edition: + page_url = urllib.parse.urljoin(public_base, md_to_url_path(entry.md).lstrip("/")) + parts = exclude_parts(doc, article, page_url) + if parts: + excluded[entry.md] = parts section_id = url_to_section_id(md_to_url_path(entry.md)) section = doc.new_tag("section", attrs={"class": "print-page", "id": section_id}) for child in list(article.children): section.append(child.extract()) namespace_ids(section, section_id) - unlink_ids = dropped_ids if any(entry.md.startswith(prefix) for prefix in shortened) else frozenset() + unlink_ids = dropped_ids if any(rule.matches(entry.md) for rule in shortening) else frozenset() resolve_links(section, md_to_url_path(entry.md), valid_ids, public_base, unlink_ids) demote_headings(section, entry.depth) if entry.depth == 0: part_cover(doc, section) doc.body.append(section) - return doc, missing + return doc, missing, excluded # -- normalization ------------------------------------------------------------- @@ -1132,11 +1192,11 @@ def build_pdf( entries, dropped = apply_section_rules(entries, rules) for rule in rules: if rule.mode != "full": - count = sum(1 for md in dropped if md.startswith(rule.prefix)) + count = sum(1 for md in dropped if rule.matches(md)) print(f"Print edition: {rule.prefix} as {rule.mode}, {count} pages dropped") config = load_site_config() public_base = public_base_url(config, version) - doc, missing = merge_pages(entries, SITE_DIR, public_base, rules, dropped) + doc, missing, excluded = merge_pages(entries, SITE_DIR, public_base, rules, dropped, print_edition) pages = sum(1 for e in entries if e.md) - len(missing) headings = sum(1 for e in entries if is_heading_entry(e)) if missing: @@ -1146,6 +1206,8 @@ def build_pdf( + (" ..." if len(missing) > 5 else ""), file=sys.stderr, ) + for md, count in sorted(excluded.items()): + print(f"Print edition: {count} {'part' if count == 1 else 'parts'} of {md} left out") print(f"Merged {pages} pages and {headings} section headings along {NAV_YML}") stats, warnings = normalize(doc, SITE_DIR, work_dir / "images" if print_edition else None) diff --git a/tools/pdf/print.yml b/tools/pdf/print.yml index 20c0b6e40..80955db36 100644 --- a/tools/pdf/print.yml +++ b/tools/pdf/print.yml @@ -50,17 +50,23 @@ authors: # darker in black and white. body-weight: light -# How a navigation section is printed, keyed by the docs/ directory its pages -# live in. A section that is not listed prints in full. +# How a navigation section or a page is printed, keyed by the docs/ directory +# its pages live in (ending in /) or by a single page (ending in .md). What is +# not listed prints in full; keys do not nest. # full - as on the site -# list - its overview: the section page and the overview pages of its -# subsections; pages no overview lists become a two-column table of -# title and first paragraph, under `columns` when given; navigation -# titles whose pages an overview lists become one table of the -# titles and their page names -# omit - left out; a note names the online edition +# list - a directory only: its overview - the section page and the overview +# pages of its subsections; pages no overview lists become a +# two-column table of title and first paragraph, under `columns` +# when given; navigation titles whose pages an overview lists become +# one table of the titles and their page names +# omit - left out without a trace; the imprint names the online edition as +# the complete reference. The only mode a page accepts +# A part of a page is left out with the class `print-exclude` in the Markdown, +# e.g. ```sql { .print-exclude }; a note naming the page online takes its place. sections: build/reference/: list release-notes/: mode: list columns: [Release, Summary] + develop/cmem-client-api/: omit + build/tutorial-how-to-link-ids-to-osint/: omit From 591da053d9a60238b147e53524280c65694057a1 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Tue, 15 Sep 2026 16:54:54 +0200 Subject: [PATCH 04/15] plan for review findings. --- tasks/backlog.md | 109 +++++++++++++++++++++++- tasks/spec.md | 218 +++++++++++++++++++++++++++++++++++++++++++++-- 2 files changed, 317 insertions(+), 10 deletions(-) diff --git a/tasks/backlog.md b/tasks/backlog.md index 9223161ae..4f9c51087 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -13,7 +13,8 @@ Every task names how it is verified. A task is not done until that verification Spec §4, D1-D11: BoD, A4, black and white on 80 g, no ISBN, authors by name (first GitHub IDs, revised 2026-09-15), separate screen and print editions, section modes with A.3 and Release Notes as lists, page references and URL footnotes, no logo or version on text pages, 10 pt body, optional Ghostscript normalization. D12-D14, decided -2026-09-15: excluding content from the print edition (spec §10), implemented by P18. +2026-09-15: excluding content from the print edition (spec §10), implemented by P18. D15-D18, decided +2026-09-15: the findings of the pull request review (spec §11), backlog P19-P24. --- @@ -316,11 +317,13 @@ pages blank. Non-zero exit on a violation; run by `task pdf:print`, on the norma ## P15 - Low-resolution originals -49 images are below 150 ppi at printed size; resampling (P12) hides that from the preflight but adds no -detail. Work through the list: replace with a fresh screenshot, or accept in spec §8. +49 images were below 150 ppi at printed size; resampling (P12) hides that from the preflight but adds no +detail. After P18, 43 images in the printed pages that are not generated print below 150 ppi, 37 of them +without a `width` - P24 sizes those first (spec §11). Work through what stays below 150 ppi: replace it +with a fresh screenshot, or accept it in spec §8. **Verify:** every entry replaced or accepted. -**Est:** medium, mostly content. **Depends on:** P12. +**Est:** medium, mostly content. **Depends on:** P12, P24. --- @@ -398,6 +401,102 @@ from the stamp. `task check` passes. --- +## P19 - Part label in the footer + +Spec §11: the running footer of a left-hand page prints the part as the part band does, `Part A: Build`. +The contents, the bookmarks and the right-hand footer keep theirs. `print-footer` in `tools/pdf/style.typ`. + +**Verify:** render a left-hand and a right-hand page of two parts; the screen PDF is unchanged. +**Est:** small. **Depends on:** P3. + +--- + +## P20 - Web addresses as endnotes + +Spec §11, D15. In the print edition, a link out of the book prints its text and a superscript number +instead of a footnote. The numbers run within a part and start again at 1 in the next; an address cited +twice in a part keeps its first number. A list "Web addresses" on a new page, under an unnumbered heading, +closes each part that cites any: number, address and the pages citing it, laid out without link +annotations. Links within the book keep their page reference. + +**Verify:** no footnote left in the book block; each part's list holds every number of that part with its +address and correct pages; numbering restarts per part; no link annotations from notes; Typst reports no +layout that failed to converge; compile time and page count before and after. +**Est:** medium. **Depends on:** P9. + +--- + +## P21 - Author order from the printed content + +Spec §11, D16. `dec-tool pdf-authors` counts only the commits to the files the print edition +prints: the pages left after the section rules and the images in their directories, from +`git log --no-merges` following renames, each commit once, generated pages left out. The GitHub +commits API maps each commit to its account. Bots, agents, anonymous commits and `authors.exclude` +stay excluded; the imprint says "most commits to the printed pages first". + +**Verify:** unit tests with a fake history for file selection, rename following, counting and account +mapping; a real run shows the new order next to today's; the imprint follows it. +**Est:** medium. **Depends on:** P6, P18. + +--- + +## P22 - Cards of equal height + +Spec §11: `cards()` in `tools/pdf/style.typ` lays out its grid row by row and gives both cards of a row +the height of the taller one; a card alone in the last row keeps its own height. The rounded frame and +`breakable: false` stay; the screen PDF changes with it. + +**Verify:** render the card grids of the part pages in both editions; facing cards end on one line; no +card breaks across pages. +**Est:** small. **Depends on:** nothing. + +--- + +## P23 - Operator reference in a compact format + +Spec §11, D17. A section mode `reference` prints `build/reference/` as one compact, harmonized +entry per operator; the entries replace the overview tables. + +- **Structure:** each type chapter keeps its introduction. Operators follow alphabetically, numbered + A.3.x.y and listed in the part contents; the transformer category moves into the entry. +- **Entry:** the title and a field line with type, category, plugin ID, `Python plugin` and a distance + range. Then the rendered description without `## Examples` and without the Python plugin note, its + headings as run-in labels. Then one parameter table with the columns Parameter and its ID, Type, Default + and Description, an `Advanced` row and `parent.child` sub-parameters, and a `Related:` line with page + references. +- **Sparse data prints as nothing:** no table without parameters, `–` for a missing default, and + `see below` with a code block for a multi-line default. The data types map to the vocabulary of spec + §11. +- **Data:** structure and parameters come from `data/plugins.json`, the description from the site page. + The build fails when pages and JSON disagree. +- **Configuration:** `print.yml` switches `build/reference/` from `list` to `reference`. The A.3 + introduction names the online examples and Python plugins once. + +**Verify:** unit tests build entries from a JSON fixture - an operator without parameters, one with +advanced and sub-parameters, a multi-line default, every data type - and cover the page/JSON check. The +print build shows 389 entries, no Examples heading and no `None`. The page count is measured against the +estimate of spec §11 (about 860) and BoD's limit (P14). Render a transformer, a dataset and a custom task +with more than 20 parameters. +**Est:** medium to large. **Depends on:** P8. + +--- + +## P24 - Image widths in the sources + +Spec §11, D18. `dec-tool image-widths --check|--fix` writes `{ width="NN%" }` for raster images without a +width in pages that are not generated: the pixel width divided by the capture scale and by the full page +width - the 16 cm text column, 605 CSS pixels - rounded to 5 %, at most 100 %. The capture scale is 2 for +144 dpi (macOS Retina), otherwise the declared density / 96, and 1 without one. Optionally `task check` +runs `--check`. + +**Verify:** unit tests for capture scale, percentage, rounding and the skip rules; after `--fix` no raster +image in a page that is not generated lacks a width; spot checks of the changed pages on the site and in +the print edition; the count below 150 ppi in the print edition before and after - no image prints less +sharply than before. +**Est:** medium, touches many pages. **Depends on:** nothing. P15 follows it. + +--- + ## Sequencing ```text @@ -411,6 +510,8 @@ P1 ──> P2 ──> P3 ──> P4 ──> P5 ──> P7 └──> P14 ──> P16 P17 after P8 and P14 P18 after P8 and P9 +P19 after P3, P20 after P9, P21 after P6 and P18, P22 any time, P23 after P8 +P24 before P15 ``` P1, P6, P8, P10, P11 and P12 can start now. diff --git a/tasks/spec.md b/tasks/spec.md index 48ab3266b..6af102e0c 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -3,7 +3,8 @@ **Status:** accepted 2026-09-14 - all decisions made (§4). Backlog P0-P12 implemented 2026-09-15 (`task pdf:print`, 962 pages); P13-P17 open. The backlog's "Done" notes record where the implementation refines this spec. §10, excluding content from the print edition, was decided on 2026-09-15 (D12-D14) -and implemented the same day (backlog P18): the print edition has 666 pages. +and implemented the same day (backlog P18): the print edition has 666 pages. §11 collects the pull request +review of 2026-09-15, decided the same day (D15-D18); backlog P19-P24. **Branch:** `feature/print-on-demand`, based on `main` at `c20d74b94` (PDF export merged). **Goal:** a *book block* - the interior file of a printed, perfect-bound book - built next to the screen PDF, which BoD accepts without rework. @@ -70,7 +71,7 @@ For comparison, not pursued: Amazon KDP allows 828 pages (black ink on white), L - Page numbers sit on the **outer edge**: left on even (verso) pages, right on odd (recto) pages. - Running titles mirror too, following book convention (the larger unit on the left page): - - verso: `1234` at the outer left, `A Build` beside it + - verso: `1234` at the outer left, `Part A: Build` beside it (revised after the review, §11) - recto: `A.3 Task and Operator Reference` beside `1235` at the outer right - `| total` is dropped - it has no meaning on paper. - No logo and no version stamp on text pages (D9); both stay on the title page and the imprint. @@ -153,29 +154,34 @@ in `authors.names`, or they print as their ID, and the build warns about each. ### R4 - Links on paper In the print edition (D8): internal links print their text plus a page reference `(p. 34)`; external -links print their text with a footnote holding the URL; no colour, underline or arrow. Link annotations +links print their text with a superscript number, and each part ends with a list of its web addresses +(D15, §11); no colour, underline or arrow. Link annotations are left out of the book block - Ghostscript refuses PDF/X output while a page carries one (§7). ## 4. Decisions -D1-D11 made 2026-09-14; D12-D14 made 2026-09-15 (§10). +D1-D11 made 2026-09-14; D12-D18 made 2026-09-15 (§10, §11). | # | Question | Decision | | :-- | :-- | :-- | | D1 | Scope | The print edition reduces sections instead of printing them in full, **configurable per section** as `full`, `list` or `omit`. Default: A.3 Task and Operator Reference → `list`, H Release Notes → `list`, everything else `full` (§5) | | D2 | Provider | **BoD** (§2) | | D3 | ISBN | none for now; may come later | -| D4 | Authors | names, most commits first: from `authors.names` in `print.yml`, else the GitHub profile, else the ID; no anonymous entries, bots, agents or excluded IDs (§3). Revised 2026-09-15 - first decided as GitHub IDs | +| D4 | Authors | names, most commits first: from `authors.names` in `print.yml`, else the GitHub profile, else the ID; no anonymous entries, bots, agents or excluded IDs (§3). Revised 2026-09-15 - first decided as GitHub IDs; commits count only on printed pages (D16) | | D5 | Colour and paper | black and white interior on 80 g paper. BoD offers 80 g to publishers only - eccenca GmbH needs a publisher account | | D6 | Editions | separate screen and print PDFs. `task pdf` stays the screen edition, unchanged; `task pdf:print` builds the book block, switched by a Typst input (`edition=print`) and a print configuration | | D7 | Trim size | A4 | -| D8 | Links on paper | page references and URL footnotes, no link styling (R4) | +| D8 | Links on paper | page references and URL footnotes, no link styling (R4); the footnotes become endnotes per part (D15) | | D9 | Logo and version in the running header | title page and imprint only (R1) | | D10 | Body type size | keep 10 pt and tighten the spacing (§5). A smaller body, 9 pt or even 8 pt, only if the page limit is still exceeded - not needed for the default configuration | | D11 | Normalization | an **optional** Ghostscript pass after Typst: PDF/X-4, CMYK, all images at 300 dpi (§7) | | D12 | Subtrees and pages | path rules in `tools/pdf/print.yml`: a `sections` key may name a page, which accepts `omit` only; no front matter property, no `.pdfexclude` (§10) | | D13 | Parts of a page | the class `print-exclude`, effective in the print edition only; the site and the screen PDF are unchanged (§10) | | D14 | What stands in for excluded content | pages and subtrees: nothing, also for a directory's `omit` (revises §5); parts of a page: one note per run of parts, pointing to the page in the online edition for the full details (§10) | +| D15 | Web addresses on paper | endnotes: a superscript number per address, numbered within each part; a list "Web addresses" with each address and the pages citing it closes each part (§11; revises R4 and D8) | +| D16 | Author order | commits to the printed pages and their images, renames followed, each commit once; merges and generated pages do not count (§11) | +| D17 | Operator reference | section mode `reference`: one compact entry per operator from `data/plugins.json` and the rendered description, without examples, replacing the overview tables; operators in alphabetical order with the category as a field, listed in the part contents; one type vocabulary (§11) | +| D18 | Screenshot widths | written into the sources for raster images without a width: pixel width / capture scale / the full page width (the 16 cm text column), rounded to 5 %, at most 100 %; capture scale 2 for 144 dpi, otherwise the declared density / 96, 1 without one (§11) | ## 5. Page budget @@ -514,3 +520,203 @@ Several blocks - also inside a list item, a content tab or an admonition. `.claude/docs-guidelines/repo-conventions.md` on the class, including that generated pages cannot carry it. - **Effort:** small to medium; backlog P18. + +## 11. Review of the first print build + +**Status:** findings of the pull request review, 2026-09-15, decided the same day as D15-D18 (§4). + +| Finding | Today | Proposal | Backlog | +| :-- | :-- | :-- | :-- | +| A left-hand page's footer should read `Part A: Build` | `A Build` (R1) | the part label the part band prints | P19 | +| Web addresses belong at the end of the document | a footnote for each link out of the book (R4) | endnotes, listed at the end of each part (D15) | P20 | +| The author order should follow the printed content | commits to the whole repository (§3) | commits to the printed pages (D16) | P21 | +| Facing cards in a two-column grid should be equally high | each card as high as its text | one height per row | P22 | +| The operator reference should print descriptions and parameters | overview tables, `list` mode (§5) | one compact entry per operator, replacing the overview tables (D17) | P23 | +| PDF/X-4 and CMYK | specified, not built (§7) | unchanged | P13 | +| Low-resolution images mostly lack `width` in the Markdown | an image without `width` prints at its declared density | widths computed from pixels and capture density against the full page width, written into the sources (D18) | P24, then P15 | + +### Footer: the part label + +R1 names the part beside the page number of a left-hand page as `A Build`. The part band already prints +`Part A: Build`, while the contents and the bookmarks print `A Build`. The left-hand footer follows the +band: `1234 Part A: Build`. The contents, the bookmarks and the right-hand footer stay as they are. + +### Web addresses as endnotes (D15) + +Today every link out of the book prints its address in a footnote on the same page (R4). Decided: + +- A link out of the book prints its text and a superscript number. +- The numbers run within a part and start again at 1 in the next one. An address cited more than once in + a part keeps its first number there. +- A list "Web addresses", on a new page under an unnumbered heading, closes each part that cites any. It + holds each number, its address and the pages that cite it, for example + `17 https://github.com/eccenca/cmemc (pp. 412, 530)`. +- The list is laid out without link annotations, like the contents (P9). Footnotes link marker and entry + internally, which Typst cannot switch off, and PDF/X output wants no annotation at all (§7). +- Links within the book keep their page reference. + +Numbering in a `context` rule over every link costs layout passes; measure the compile time and watch +for Typst's warning that the layout did not converge. + +### Author order from the printed content (D16) + +Today `tools/pdf/authors.yml` counts a contributor's commits to the whole repository, from the GitHub +contributors API. Tooling, generated references and content the print edition leaves out weigh as much +as the printed pages. The repository has 1,633 commits, 1,292 of them touching `docs/`. Decided: + +- The printed files: the pages the print edition prints after the section rules (§5, §10), and the + images in their directories. +- The commits: `git log --no-merges` for those files, following renames. A commit counts once, however + many printed files it touches. +- The account of each commit: `author.login` from the GitHub commits API, which maps a commit's e-mail + to a GitHub account - one request per 100 commits, about 17. +- The existing rules stay: no anonymous commits, bots or agents; `authors.exclude`; names from + `authors.names` or the profile. +- `authors.yml` records these counts. The imprint reads "most commits to the printed pages first". + +Commits to generated pages credit whoever ran the generator, so they do not count. + +### Cards of equal height + +`cards()` sets `card()` blocks in two columns, each as high as its text, so facing cards end at +different heights. Plan: lay the grid out row by row, measure both cards of a row at the column +width, and give both the taller height. A card alone in the last row keeps its own height. The rounded +frame and `breakable: false` stay. The screen PDF uses the same grid and changes with it. + +### The operator reference in a compact format (D17) + +A.3 prints as `list` (§5): the overview pages with their `Name | Description` tables, while the 389 +operator pages are dropped. The review asks for the operators themselves, with their descriptions and +parameters but without examples. In the print edition the operator entries replace the overview tables. +The format below was decided as proposed (D17). + +All 389 operators share one structure. `tools/templates/plugin.md` generates their pages, and the +generator dumps the same data to `data/plugins.json`, which is tracked and was last regenerated together +with the pages on 2026-09-02. Much of the structure is sparse (measured 2026-09-15): + +| Part | Measured | +| :-- | :-- | +| parameters | 1,040 in all; 59 operators have none, 316 have no advanced ones. Median per operator: 4 for custom workflow tasks (at most 24), 3 for datasets, 1 for distance measures and transformers, 0 for aggregators | +| defaults | 431 parameters have none; 8 are multi-line, 4 structured | +| parameter descriptions | 27 are empty | +| sub-parameters | 21 | +| descriptions | median 263 characters; 154 under 200, 43 over 2,000 | +| examples | on 91 operators, 19 % of the documentation text | +| related plugins | on 81 operators | +| Python plugins | 71 | + +A generated page spends a heading and three bullets on every parameter (`ID`, `Datatype`, +`Default Value`). It writes `None` both for a missing default and for an empty advanced section, and +repeats the Python plugin note on every Python operator. + +#### Structure + +- **Chapters:** each operator type keeps its chapter and its introduction, A.3.1 Aggregators to A.3.5 + Transformers. The chapter's operator entries replace its overview table. +- **Order:** operators follow in alphabetical order, as in the overview tables. The transformer category + becomes a field of the entry instead of a level of headings. So every operator sits at the same level, + A.3.x.y, and the part contents list them with their pages. +- **Introduction:** the introduction of A.3 says once what an entry shows, that the examples are part of + the online edition, and what a Python plugin needs. + +#### An entry + +```text +A.3.5.42 Constant transformer · Value · constant + +Generates a constant value. + +Parameter Type Default Description +Value text – The constant value to be generated +value +``` + +- **Heading:** the operator's title, numbered, kept together with what follows. +- **Field line:** the operator type, the transformer category, the plugin ID in monospace, `Python plugin` + where it applies, and a distance measure's range. It is small, grey and one line long. +- **Description:** the rendered description from the site page, without its `## Examples` section and + without the Python plugin note. Its own second-level headings, such as `## Characteristics`, print as + run-in labels, not as numbered sections. +- **Parameters:** one table per operator, in the plugin's order. The columns are Parameter (the title, and + the ID in monospace below it), Type, Default and Description. Advanced parameters follow in the same + table after a row labelled `Advanced`; a sub-parameter follows its parameter as `parent.child`. +- **Related:** one line, for example `Related: Merge (p. 214), Zip (p. 230)`. + +Sparse data prints as nothing: + +- An operator without parameters has no table, and one without advanced parameters no `Advanced` row. +- A parameter without a default shows `–`; a password never shows one. +- A multi-line or structured default shows `see below`, and a code block follows the table. +- An operator without related plugins has no `Related` line, and an empty parameter description leaves + its cell empty. + +The data types print in one vocabulary: + +| Data type | Printed | +| :-- | :-- | +| `string`, `multiline string` | text | +| `int`, `Long` | integer | +| `double` | number | +| `boolean` | boolean | +| `char` | character | +| `enumeration` | choice | +| `password` | password | +| `resource` | file | +| `scheme:string` | URI - to confirm | +| `traversable[string]` | list of text | +| `stringmap` | map | +| `code-sparql` | SPARQL | + +#### Data source + +The build takes the structure and the parameters from `data/plugins.json` and only the description from +the rendered site page. The JSON tells a missing default apart from the text `None`, and it carries the +types, the advanced flags and the sub-parameters that the page shows only as bullets. The build fails when +an operator page and the JSON disagree - an operator without an entry, or an entry without a page - so a +regeneration that updated only one of them shows. + +#### Budget + +A rough count - 95 characters per line of description, one row per parameter and more for a long +description, four lines for heading and field line - gives about 180 pages for the 389 entries. The part +contents add about 8 pages. The book would grow from 666 to about 860 pages; the operator pages in full +took about 458 (§5). BoD takes 1,200 pages on 80 g and 1,050 on 90 g (D5). P23 measures the real count. + +The site and the generator stay as they are. The same format could later serve the site as well; that is +not part of this proposal. + +### PDF/X-4 and CMYK + +Unchanged: the optional normalization pass (§7, D11), backlog P13. The review confirms that it is +needed. + +### Image widths in the sources (D18) + +Typst prints an image without `width` at the pixel density it declares, 72 dpi when it declares none, +and never wider than the column. On 2026-09-15 the printed pages that are not generated hold 500 raster +images, 352 of them without `width`. The declared densities: none on 231, 144 dpi on 180 (Retina +captures on macOS), 120 dpi on 80, 96 dpi on 8, 192 dpi on 1. 43 images print below 150 ppi, 37 of them +without `width`. + +Decided (D18) - `dec-tool image-widths`, with `--check` and `--fix`, writes the missing `{ width="NN%" }` +into the Markdown of pages that are not generated: + +- **Capture scale:** 2 for 144 dpi, the density of a Retina capture on macOS (2 × 72). Any other declared + density is divided by 96, so 120 dpi gives 1.25 and 192 dpi gives 2. An image that declares none, 72 or + 96 dpi has scale 1. +- **Natural width:** the pixel width divided by the capture scale, in CSS pixels at 96 per inch. +- **Width:** the natural width as a share of the full page width - the 16 cm text column, 605 CSS pixels - + rounded to 5 % and at most 100 %. +- **Scope:** raster images only. SVGs and images that already have a width stay as they are. +- **Check:** `task check` may run `--check`, so a new screenshot gets its width. + +The same percentage serves the site and the PDF. In print an image then keeps its natural size and prints +at its capture scale times 96 ppi - 96 ppi at scale 1, 192 ppi for a Retina capture - and an image wider +than the column still fills it. No image prints less sharply than today, when Typst assumes 72 dpi for an +image without a declared density. P15 handles what stays below 150 ppi. + +### Decisions + +Decided on 2026-09-15 as D15-D18, recorded in §4: web addresses in a list at the end of each part (D15), +the author order from commits to the printed pages (D16), the compact operator reference as proposed +(D17), and screenshot widths against the full page width (D18). From 4a3c3de7f4f8afe1197f98aa0fb1101843ca746a Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Tue, 15 Sep 2026 17:29:14 +0200 Subject: [PATCH 05/15] implement the review findings of the print edition - left-hand footers name the part as "Part A: Build" - web addresses become per-part endnote lists instead of footnotes - the author order counts only commits to printed pages and images - facing cards in a two-column grid share one height - print the operator reference as compact entries from data/plugins.json: fields, description without examples, one parameter table and related operators, replacing the overview tables The print edition grows from 666 to 870 pages; the screen PDF is unchanged. --- Taskfile.yml | 9 +- tasks/backlog.md | 63 +++++- tasks/spec.md | 7 +- tests/test_build_pdf_print.py | 16 ++ tests/test_build_pdf_reference.py | 215 ++++++++++++++++++ tests/test_pdf_authors.py | 89 ++++++++ tools/README.md | 2 +- tools/build_pdf.py | 361 +++++++++++++++++++++++++++++- tools/pdf/authors.yml | 61 +++-- tools/pdf/filter.lua | 5 + tools/pdf/print.yml | 5 +- tools/pdf/style.typ | 125 +++++++++-- tools/pdf_authors.py | 138 ++++++++++-- 13 files changed, 1006 insertions(+), 90 deletions(-) create mode 100644 tests/test_build_pdf_reference.py diff --git a/Taskfile.yml b/Taskfile.yml index ad056052a..64fea8820 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -228,10 +228,11 @@ tasks: pdf:authors: desc: Refresh the author list of the print edition's imprint summary: | - Writes tools/pdf/authors.yml from the GitHub contributors of this - repository and the names their GitHub profiles show: most commits - first, without anonymous contributions, bot accounts, software agents - and the IDs excluded in tools/pdf/print.yml. Adds each author that + Writes tools/pdf/authors.yml from the commits to what the print edition + prints - its pages that are not generated, and the images they + reference: most commits first, with the names the GitHub profiles show, + without anonymous commits, bot accounts, software agents and the IDs + excluded in tools/pdf/print.yml. Adds each author that print.yml does not list yet to its authors.names, without a name, and warns about each author who would print as a GitHub ID. Commit both files. Uses GITHUB_TOKEN or GH_TOKEN, else the token of a logged-in diff --git a/tasks/backlog.md b/tasks/backlog.md index 4f9c51087..bd60b1783 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -401,7 +401,7 @@ from the stamp. `task check` passes. --- -## P19 - Part label in the footer +## P19 - Part label in the footer - **done** Spec §11: the running footer of a left-hand page prints the part as the part band does, `Part A: Build`. The contents, the bookmarks and the right-hand footer keep theirs. `print-footer` in `tools/pdf/style.typ`. @@ -409,9 +409,13 @@ The contents, the bookmarks and the right-hand footer keep theirs. `print-footer **Verify:** render a left-hand and a right-hand page of two parts; the screen PDF is unchanged. **Est:** small. **Depends on:** P3. +**Done 2026-09-15:** `print-footer` prints a part on a left-hand page as `Part A: Build`; right-hand footers, +contents and bookmarks keep `A Build`. 313 left-hand pages carried the label in the build before P23; +the screen PDF is unchanged. + --- -## P20 - Web addresses as endnotes +## P20 - Web addresses as endnotes - **done** Spec §11, D15. In the print edition, a link out of the book prints its text and a superscript number instead of a footnote. The numbers run within a part and start again at 1 in the next; an address cited @@ -424,9 +428,19 @@ address and correct pages; numbering restarts per part; no link annotations from layout that failed to converge; compile time and page count before and after. **Est:** medium. **Depends on:** P9. +**Done 2026-09-15:** + +- `web-address()` in `style.typ` places the address as metadata and prints a superscript number, counted + within the part with `context` and `query`. `part-addresses()` sets the list on a new page under an + unnumbered heading that is bookmarked but not in the contents: number, address and the pages citing it. +- In the print edition, `merge_pages` puts a `part-end` marker at the end of every part, and + `filter.lua` turns it into `#part-addresses()`; a unit test covers the markers. +- Measured with P23 in place: 8 lists with 390 entries, 159 of them in part A. Typst compiles in 4 s + without a convergence warning; the screen PDF is unchanged. + --- -## P21 - Author order from the printed content +## P21 - Author order from the printed content - **done** Spec §11, D16. `dec-tool pdf-authors` counts only the commits to the files the print edition prints: the pages left after the section rules and the images in their directories, from @@ -438,9 +452,23 @@ stay excluded; the imprint says "most commits to the printed pages first". mapping; a real run shows the new order next to today's; the imprint follows it. **Est:** medium. **Depends on:** P6, P18. +**Done 2026-09-15:** + +- `tools/pdf_authors.py` counts the commits to the printed files. `printed_pages` applies the section + rules, and `printed_files` returns the pages that are not generated plus the images they reference. + `file_commits` runs `git log --no-merges --follow` per file, and `commit_accounts` with + `count_commits` maps each commit once to its account, falling back to the author e-mail. + `fetch_commits` replaces the contributors API. +- The imprint reads "most commits to the printed pages first". +- `tests/test_pdf_authors.py` has 31 tests; new are printed files, counting and rename following in an + isolated git repository. +- Measured: 661 commits to 638 printed files, 18 authors instead of 19 - `haschek` has no commit to + printed content. The order starts with `rpietzsch` (249), `seebi` (156), `sobo` (46), + `muddymudskipper` (45) and `irangareddy` (37). + --- -## P22 - Cards of equal height +## P22 - Cards of equal height - **done** Spec §11: `cards()` in `tools/pdf/style.typ` lays out its grid row by row and gives both cards of a row the height of the taller one; a card alone in the last row keeps its own height. The rounded frame and @@ -450,9 +478,14 @@ the height of the taller one; a card alone in the last row keeps its own height. card breaks across pages. **Est:** small. **Depends on:** nothing. +**Done 2026-09-15:** `cards()` measures the cards of each row at the column width and frames both with +`card-frame()` at the height of the taller one; `card()` now only passes the content through. Checked on +the card grids of the Build page (p. 11) and the reference (p. 30); the screen PDF keeps its page count +and text. + --- -## P23 - Operator reference in a compact format +## P23 - Operator reference in a compact format - **done** Spec §11, D17. A section mode `reference` prints `build/reference/` as one compact, harmonized entry per operator; the entries replace the overview tables. @@ -479,6 +512,26 @@ estimate of spec §11 (about 860) and BoD's limit (P14). Render a transformer, a with more than 20 parameters. **Est:** medium to large. **Depends on:** P8. +**Done 2026-09-15:** + +- Section mode `reference` in `tools/build_pdf.py`. `reference_section` places a note after the section + page and the operators of each type after its overview page, alphabetically, drops the category + headings, and fails when pages and `data/plugins.json` disagree. `merge_pages` removes the overview + tables. +- `operator_entries` renders each entry: `operator_fields`; `operator_description` without the title, + the Python plugin note and the template sections, with headings as run-in labels; `parameter_table` + with the type vocabulary of `data_type`, an `Advanced` row, `parent.child` sub-parameters, `see below` + defaults and Markdown descriptions; and `operator_related`. +- `operator-fields()` in `style.typ` and its mapping in `filter.lua`. `print.yml` switches + `build/reference/` to `reference`. `tests/test_build_pdf_reference.py` has 5 tests. +- Measured: 389 entries with bookmarks; A.3 takes 213 pages (pp. 29-241) and the book 870 (spec §11 + estimated 860). No `Advanced Parameter` heading and no generated example is left; 9 defaults print as + `see below`, 80 entries have a `Related:` line, and 1 web address still points to an operator page. + The screen PDF is unchanged. +- Open: example subsections the plugins write into their own documentation - `5. Example` in Pivot, + `6. Examples` in RDF file, `Example usage` in Knowledge Graph - still print as run-in labels with their + text; only the generated `## Examples` sections are dropped. + --- ## P24 - Image widths in the sources diff --git a/tasks/spec.md b/tasks/spec.md index 6af102e0c..2bf6a81f4 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -4,7 +4,8 @@ (`task pdf:print`, 962 pages); P13-P17 open. The backlog's "Done" notes record where the implementation refines this spec. §10, excluding content from the print edition, was decided on 2026-09-15 (D12-D14) and implemented the same day (backlog P18): the print edition has 666 pages. §11 collects the pull request -review of 2026-09-15, decided the same day (D15-D18); backlog P19-P24. +review of 2026-09-15, decided the same day (D15-D18); backlog P19-P24, of which P19-P23 were implemented +on 2026-09-15: the print edition has 870 pages. **Branch:** `feature/print-on-demand`, based on `main` at `c20d74b94` (PDF export merged). **Goal:** a *book block* - the interior file of a printed, perfect-bound book - built next to the screen PDF, which BoD accepts without rework. @@ -524,6 +525,7 @@ Several blocks - also inside a list item, a content tab or an admonition. ## 11. Review of the first print build **Status:** findings of the pull request review, 2026-09-15, decided the same day as D15-D18 (§4). +P19-P23 are implemented; P24 and P13 are open. | Finding | Today | Proposal | Backlog | | :-- | :-- | :-- | :-- | @@ -680,7 +682,8 @@ regeneration that updated only one of them shows. A rough count - 95 characters per line of description, one row per parameter and more for a long description, four lines for heading and field line - gives about 180 pages for the 389 entries. The part contents add about 8 pages. The book would grow from 666 to about 860 pages; the operator pages in full -took about 458 (§5). BoD takes 1,200 pages on 80 g and 1,050 on 90 g (D5). P23 measures the real count. +took about 458 (§5). BoD takes 1,200 pages on 80 g and 1,050 on 90 g (D5). Measured after P23: A.3 takes 213 pages and the +book 870. The site and the generator stay as they are. The same format could later serve the site as well; that is not part of this proposal. diff --git a/tests/test_build_pdf_print.py b/tests/test_build_pdf_print.py index 554957286..d4c4cd1ae 100644 --- a/tests/test_build_pdf_print.py +++ b/tests/test_build_pdf_print.py @@ -320,6 +320,22 @@ def test_the_print_edition_replaces_each_run_of_excluded_parts_with_one_note(tmp assert missing == [] +def test_the_print_edition_marks_the_end_of_each_part(tmp_path): + site = tmp_path / "site" + write_page(site, "build/index.md", '

Build

') + write_page(site, "explore/index.md", '

Explore

') + entries = [NavEntry(0, "build/index.md"), NavEntry(0, "explore/index.md")] + + def markers(print_edition): + doc, _, _ = merge_pages(entries, site, "https://example.org/26.2/", print_edition=print_edition) + classes = [div.get("class") for div in doc.body.find_all("div")] + names = [" ".join(c) if isinstance(c, list) else c for c in classes] + return [name for name in names if name in ("chapter-break", "part-end")] + + assert markers(True) == ["chapter-break", "part-end", "chapter-break", "part-end"] + assert markers(False) == ["chapter-break", "chapter-break"] + + def test_the_screen_edition_keeps_the_excluded_parts(tmp_path): site = tmp_path / "site" write_page(site, "build/snowflake/index.md", '

Connect

On screen.

') diff --git a/tests/test_build_pdf_reference.py b/tests/test_build_pdf_reference.py new file mode 100644 index 000000000..5525fb1d6 --- /dev/null +++ b/tests/test_build_pdf_reference.py @@ -0,0 +1,215 @@ +"""Test the compact operator reference of the print edition (tasks/spec.md, §11, D17)""" +import click +import pytest + +from tools.build_pdf import ( + Generated, + NavEntry, + SectionRule, + apply_section_rules, + load_section_rules, + md_to_built_html, + merge_pages, +) + +RULE = SectionRule("build/reference/", "reference") +PUBLIC = "https://example.org/26.2/" + + +def operator(plugin_id, plugin_type="transformer", title=None, category="Value", **extra): + """A plugin description as data/plugins.json holds it, filled in only where a test needs it.""" + record = { + "pluginId": plugin_id, + "title": title or plugin_id, + "pluginType": plugin_type, + "main_category": category, + "categories": [category], + "description": f"{plugin_id} description", + "markdownDocumentation": None, + "backendType": "scala", + "distanceMeasureRange": None, + "properties": {}, + "properties_advanced": {}, + "relatedPlugins": [], + "is_deprecated": False, + } + record.update(extra) + return record + + +def parameter(name, title, parameter_type="string", value="", description="", advanced=False, properties=None): + return { + "name": name, + "title": title, + "description": description, + "type": "string", + "parameterType": parameter_type, + "value": value, + "advanced": advanced, + "visibleInDialog": True, + "properties": properties or {}, + } + + +def write_page(site, md, body): + path = site / md_to_built_html(md) + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(f'
{body}
') + + +# The shape of the reference in nav.yml: a page per type, transformers grouped by category. +NAV = [ + NavEntry(0, "build/index.md", "Build"), + NavEntry(1, "build/reference/index.md", "Task and Operator Reference"), + NavEntry(2, "build/reference/aggregator/index.md", "Aggregators"), + NavEntry(3, "build/reference/aggregator/max.md", "Maximum"), + NavEntry(2, "build/reference/transformer/index.md", "Transformers"), + NavEntry(3, title="Value"), + NavEntry(4, "build/reference/transformer/Value/uuid.md", "UUID"), + NavEntry(4, "build/reference/transformer/Value/constant.md", "Constant"), + NavEntry(1, "build/spark.md", "Spark"), +] +OPERATORS = [operator("uuid", title="UUID"), operator("constant", title="Constant"), operator("max", "aggregator", title="Maximum")] +CATALOG = { + "uuid": ("build/reference/transformer/Value/uuid.md", "UUID"), + "constant": ("build/reference/transformer/Value/constant.md", "Constant"), + "max": ("build/reference/aggregator/max.md", "Maximum"), +} + + +def test_reference_is_a_mode_for_a_directory_only(tmp_path): + path = tmp_path / "print.yml" + path.write_text("sections:\n build/reference/: reference\n") + assert load_section_rules(path) == [RULE] + path.write_text("sections:\n build/reference/index.md: reference\n") + with pytest.raises(click.ClickException, match="only `omit`"): + load_section_rules(path) + + +def test_the_operators_of_a_type_follow_its_overview_page_in_alphabetical_order(): + entries, dropped = apply_section_rules(NAV, [RULE], operators=OPERATORS) + assert entries == [ + NAV[0], + NAV[1], + NavEntry(1, generated=Generated("note", "build/reference/", "reference", ["build/reference/index.md"])), + NAV[2], + NavEntry(3, generated=Generated( + "operators", "build/reference/", "reference", [CATALOG["max"][0]], + operators=[OPERATORS[2]], catalog=CATALOG, + )), + NAV[4], + # The category heading is gone: the category is a field of the entry. + NavEntry(3, generated=Generated( + "operators", "build/reference/", "reference", [CATALOG["constant"][0], CATALOG["uuid"][0]], + operators=[OPERATORS[1], OPERATORS[0]], catalog=CATALOG, + )), + NAV[8], + ] + # The operator pages print, as entries - none is dropped. + assert dropped == set() + + +def test_the_build_fails_when_the_pages_and_the_operators_disagree(): + with pytest.raises(click.ClickException) as raised: + apply_section_rules(NAV, [RULE], operators=OPERATORS[1:] + [operator("zip", title="Zip")]) + assert "transformer/Value/uuid.md" in raised.value.message + assert "zip" in raised.value.message + + +def test_an_operator_entry_is_compact(tmp_path): + site = tmp_path / "site" + write_page( + site, "build/reference/transformer/index.md", + '

Transformers

Transform values.

Overview row
', + ) + write_page( + site, "build/reference/transformer/Value/constant.md", + '

Constant

' + '

Python Plugin

' + "

This operator is part of a Python Plugin Package.

" + '

Generates a constant value. See Spark.

' + '

Characteristics

Compares single values.

' + '

Examples

Returns [John].

  • Returns: John
' + '

Parameter

Value

The constant value

' + '

Advanced Parameter

None

' + '
  • UUID
', + ) + write_page(site, "build/reference/transformer/Value/uuid.md", '

UUID

Generates a UUID.

') + constant = operator( + "constant", title="Constant", backendType="python", + properties={ + "value": parameter("value", "Value", description="The constant `value`"), + "secret": parameter("secret", "Secret", "password", value="PASSWORD_PARAMETER:x"), + "connection": parameter( + "connection", "Connection", "objectParameter", value={"parameters": {"host": "localhost"}}, + properties={"host": parameter("host", "Host", value="localhost")}, + ), + }, + properties_advanced={ + "query": parameter("query", "Query", "code-sparql", value="SELECT *\nWHERE {}", advanced=True), + "count": parameter("count", "Count", "Long", value="5", advanced=True), + }, + relatedPlugins=[{"id": "uuid", "description": "Makes an identifier."}], + ) + uuid = operator("uuid", title="UUID") + entries = [ + NavEntry(2, "build/reference/transformer/index.md", "Transformers"), + NavEntry(3, generated=Generated( + "operators", "build/reference/", "reference", [CATALOG["constant"][0], CATALOG["uuid"][0]], + operators=[constant, uuid], catalog=CATALOG, + )), + ] + doc, missing, _ = merge_pages(entries, site, PUBLIC, [RULE], set(), print_edition=True) + assert missing == [] + text = " ".join(doc.body.get_text(" ").split()) + + # The overview keeps its introduction; its table gives way to the entries. + assert "Transform values." in text and "Overview row" not in text + assert [(h.name, h.get_text()) for h in doc.body.find_all(["h3", "h4", "h5"])] == [ + ("h3", "Transformers"), ("h4", "Constant"), ("h4", "UUID"), + ] + + fields = doc.body.find_all("div", class_="operator-fields") + assert [" ".join(f.get_text(" ").split()) for f in fields] == [ + "Transformer · Value · constant · Python plugin", + "Transformer · Value · uuid", + ] + + # The description: without the Python plugin note and the examples, its own + # headings as run-in labels, its links resolved like any page's. + assert "Python Plugin Package" not in text and "Returns" not in text + assert doc.body.find("strong", string="Characteristics") is not None + assert doc.body.find("a", string="Spark")["href"] == f"{PUBLIC}build/spark/" + + # One table, for Constant only: UUID has no parameters. + tables = doc.body.find_all("table") + assert len(tables) == 1 + rows = [[" ".join(c.get_text(" ").split()) for c in row.find_all(["th", "td"])] for row in tables[0].find_all("tr")] + assert rows == [ + ["Parameter", "Type", "Default", "Description"], + ["Value value", "text", "–", "The constant value"], + ["Secret secret", "password", "–", ""], + ["Connection connection", "group", "–", ""], + ["Host connection.host", "text", "localhost", ""], + ["Advanced"], + ["Query query", "SPARQL", "see below", ""], + ["Count count", "integer", "5", ""], + ] + default = doc.body.find("pre") + assert "SELECT *" in default.get_text() + assert default.find_previous("p").get_text() == "Default of query:" + + related = doc.body.find("p", class_="operator-related") + assert " ".join(related.get_text(" ").split()) == "Related: UUID" + assert related.find("a")["href"] == "#build-reference-transformer-Value-uuid" + + +def test_the_reference_opens_with_a_note_on_what_an_entry_shows(tmp_path): + site = tmp_path / "site" + write_page(site, "build/reference/index.md", '

Task and Operator Reference

') + note = Generated("note", "build/reference/", "reference", ["build/reference/index.md"]) + doc, _, _ = merge_pages([NavEntry(1, generated=note)], site, PUBLIC, [RULE], set(), print_edition=True) + text = " ".join(doc.body.get_text(" ").split()) + assert "every operator of the Task and Operator Reference with its description and parameters" in text + assert f"The examples are part of the online edition: {PUBLIC}build/reference/" in text + assert "Python plugin" in text diff --git a/tests/test_pdf_authors.py b/tests/test_pdf_authors.py index 2ada8ce97..6457c30a2 100644 --- a/tests/test_pdf_authors.py +++ b/tests/test_pdf_authors.py @@ -1,9 +1,11 @@ """Test the author list of the print edition's imprint""" +import os import shutil import subprocess import time import urllib.error import urllib.request +from pathlib import Path import click import pytest @@ -13,9 +15,13 @@ AuthorRules, add_names, authors_yaml, + commit_accounts, + count_commits, + file_commits, get_json, github_token, imprint_names, + printed_files, load_author_rules, load_imprint_names, next_page, @@ -68,6 +74,89 @@ def test_bots_agents_and_anonymous_contributions_are_not_authors(): assert select_authors(contributors) == [{"id": "sobo", "commits": 53}] +def test_printed_files_are_the_pages_not_generated_and_the_images_they_reference(tmp_path): + docs = tmp_path / "docs" + (docs / "build/tutorial").mkdir(parents=True) + (docs / "build/shared").mkdir() + (docs / "build/reference").mkdir() + (docs / "build/tutorial/index.md").write_text( + "# Tutorial\n\n" + '![Shot](shot.png){ width="50%" }\n\n' + '\n\n' + "![Remote](https://example.org/remote.png)\n\n" + "![Missing](gone.png)\n" + ) + (docs / "build/tutorial/shot.png").write_bytes(b"png") + (docs / "build/shared/logo.png").write_bytes(b"png") + (docs / "build/reference/index.md").write_text( + "# Reference\n\n\n![x](x.png)\n" + ) + files = printed_files(["build/tutorial/index.md", "build/reference/index.md", "build/missing.md"], docs) + assert [Path(f).relative_to(docs).as_posix() for f in files] == [ + "build/tutorial/index.md", + "build/tutorial/shot.png", + "build/shared/logo.png", + ] + + +def test_each_commit_counts_once_for_the_account_that_made_it(): + commits = { + "a1": "jane@example.org", + "a2": "jane@example.org", + "b1": "JOHN@example.org", + "c1": "bot@example.org", + "d1": "nobody@example.org", + } + api = [ + {"sha": "a1", "author": {"login": "jane", "type": "User"}, "commit": {"author": {"email": "jane@example.org"}}}, + {"sha": "x9", "author": {"login": "john", "type": "User"}, "commit": {"author": {"email": "john@example.org"}}}, + {"sha": "c1", "author": {"login": "dependabot[bot]", "type": "Bot"}, "commit": {"author": {"email": "bot@example.org"}}}, + {"sha": "d1", "author": None, "commit": {"author": {"email": "nobody@example.org"}}}, + ] + by_sha, by_email = commit_accounts(api) + records = count_commits(commits, by_sha, by_email) + # a2 is not on GitHub yet and b1 was made elsewhere: their e-mail names the + # account, case-insensitively. d1 maps to no account and is left out. + assert sorted(records, key=lambda r: r["login"]) == [ + {"login": "dependabot[bot]", "type": "Bot", "contributions": 1}, + {"login": "jane", "type": "User", "contributions": 2}, + {"login": "john", "type": "User", "contributions": 1}, + ] + assert [a["id"] for a in select_authors(records)] == ["jane", "john"] + + +def test_file_commits_follow_renames_and_skip_merges(tmp_path): + isolated = { + **os.environ, + "GIT_CONFIG_GLOBAL": os.devnull, + "GIT_CONFIG_NOSYSTEM": "1", + "GIT_AUTHOR_NAME": "Jane", "GIT_AUTHOR_EMAIL": "jane@example.org", + "GIT_COMMITTER_NAME": "Jane", "GIT_COMMITTER_EMAIL": "jane@example.org", + } + + def git(*args): + subprocess.run(["git", *args], cwd=tmp_path, env=isolated, check=True, capture_output=True) + + git("init", "-q", "-b", "main") + (tmp_path / "old.md").write_text("one\n") + git("add", ".") + git("commit", "-q", "-m", "one") + git("mv", "old.md", "new.md") + git("commit", "-q", "-m", "rename") + git("checkout", "-q", "-b", "side") + (tmp_path / "new.md").write_text("two\n") + git("commit", "-q", "-am", "two") + git("checkout", "-q", "main") + (tmp_path / "other.md").write_text("other\n") + git("add", ".") + git("commit", "-q", "-m", "other") + git("merge", "-q", "--no-ff", "-m", "merge", "side") + commits = file_commits([Path("new.md")], repo=tmp_path, env=isolated) + # one, rename and two - the merge commit does not count, other.md is not asked for. + assert len(commits) == 3 + assert set(commits.values()) == {"jane@example.org"} + + def test_excluded_ids_are_not_authors_whatever_their_case(): contributors = [ {"login": "seebi", "contributions": 532, "type": "User"}, diff --git a/tools/README.md b/tools/README.md index de9df3d43..8172defd8 100644 --- a/tools/README.md +++ b/tools/README.md @@ -11,7 +11,7 @@ supported entry points and pass the right options. | :------ | :------ | :------ | | `build-navigation` | `task update:navigation`, `task check:navigation` | Build `nav.yml` from the `docs/**/.pages` files; `--check` diffs instead of writing and fails on drift | | `build-pdf` | `task pdf`, `task pdf:print` | Merge the built `site/` along `nav.yml` into one document and typeset it as a single PDF with pandoc and Typst; style, filter and fonts are in `pdf/` | -| `pdf-authors` | `task pdf:authors` | Write the print edition's author list `pdf/authors.yml` from the GitHub contributors and the names their profiles show; names and exclusions are kept in `pdf/print.yml`, where it adds each new author without a name | +| `pdf-authors` | `task pdf:authors` | Write the print edition's author list `pdf/authors.yml` from the commits to the printed pages and the names on the authors' GitHub profiles; names and exclusions are kept in `pdf/print.yml`, where it adds each new author without a name | | `check-zensical-output` | `task check:output` | Inspect the built `site/` and fail if a feature we reimplemented for Zensical regressed | | `localize-bundle-assets` | `task build` | Rewrite the third-party asset URLs Zensical bakes into its JavaScript bundle to the vendored copies | | `update-icons` | `task update:icons` | Fetch the eccenca icon set from the gui-elements repository | diff --git a/tools/build_pdf.py b/tools/build_pdf.py index 88fd6d898..a32534167 100644 --- a/tools/build_pdf.py +++ b/tools/build_pdf.py @@ -22,6 +22,7 @@ import base64 import hashlib import io +import json import re import shutil import subprocess @@ -34,8 +35,9 @@ from pathlib import Path import click +import markdown import yaml -from bs4 import BeautifulSoup, NavigableString +from bs4 import BeautifulSoup, NavigableString, Tag from PIL import Image from tools.pdf_authors import load_imprint_names @@ -72,7 +74,7 @@ # The screen PDF, and the book block of the printed book (tasks/spec.md). EDITIONS = ("screen", "print") # How the print edition prints a navigation section (tools/pdf/print.yml). -SECTION_MODES = ("full", "list", "omit") +SECTION_MODES = ("full", "list", "omit", "reference") DEFAULT_COLUMNS = ("Page", "Summary") # A `groups` table: the navigation titles under an overview page - the operator # categories of the reference - and their pages, under the overview's title. @@ -82,6 +84,32 @@ PRINT_EXCLUDE = "print-exclude" EXCLUDED_NOTE = "print-excluded" PART_NOTE = "This print edition leaves out a part of this page. The online edition has the full details:" +# The operator reference in `reference` mode (tasks/spec.md, §11): the plugin +# descriptions its pages are generated from (tools/update_di_reference.py), and +# how an entry names operator types and data types. +OPERATORS_JSON = Path("data/plugins.json") +OPERATOR_TYPES = { + "aggregator": "Aggregator", + "customtask": "Workflow task", + "dataset": "Dataset", + "distancemeasure": "Distance measure", + "transformer": "Transformer", +} +DATA_TYPES = { + "string": "text", "multiline string": "text", "boolean": "boolean", "int": "integer", "Long": "integer", + "double": "number", "char": "character", "enumeration": "choice", "password": "password", + "resource": "file", "scheme:string": "URI", "uri": "URI", "graph uri": "graph URI", + "traversable[string]": "list of text", "stringmap": "map", "keyValuePairs": "key-value pairs", + "objectParameter": "group", "duration": "duration", "template": "template", "task": "task", + "project": "project", "SPARQL endpoint": "SPARQL endpoint", "identifier": "identifier", "locale": "locale", +} +CODE_LANGUAGES = { + "sparql": "SPARQL", "sql": "SQL", "json": "JSON", "yaml": "YAML", "jinja2": "Jinja template", + "python": "Python", "html": "HTML", "xml": "XML", +} +# The sections of an operator page an entry does not print: the examples, and +# what it sets from the plugin description instead. +OPERATOR_SECTIONS_LEFT_OUT = {"examples", "example", "parameter", "parameters", "advanced parameter", "related plugins"} # The print edition's text column is 16 cm wide, and BoD asks for images at # 300 dpi at their printed size (tasks/spec.md, §2). TEXT_WIDTH_PT = 16 / 2.54 * 72 @@ -95,7 +123,7 @@ class Generated: A `note` names the online edition for a section in `list` mode; a `table` lists pages that no overview page of the section lists; `groups` lists the navigation titles whose pages an overview page lists, each with the names - of its pages. + of its pages; `operators` sets the compact entries of an operator reference. """ kind: str @@ -107,6 +135,10 @@ class Generated: name: str | None = None # The rows of a `groups` table: a navigation title and the page entries beneath it. groups: list[tuple[str, list[NavEntry]]] = field(default_factory=list) + # An `operators` entry: the plugin descriptions of its pages, in their order, + # and the page and title of every operator of the reference by plugin ID. + operators: list[dict] = field(default_factory=list) + catalog: dict[str, tuple[str, str]] = field(default_factory=dict) @dataclass @@ -402,8 +434,82 @@ def omit_section(entries: list[NavEntry], rule: SectionRule) -> tuple[list[NavEn return drop_empty_headings(result), dropped -def apply_section_rules(entries: list[NavEntry], rules: list[SectionRule]) -> tuple[list[NavEntry], set[str]]: - """Shorten the navigation by the section modes. Returns the entries and the pages dropped from them.""" +def load_operators(path: Path = OPERATORS_JSON) -> list[dict]: + """The plugin descriptions of the operators the reference documents: all that are not deprecated.""" + data = json.loads(path.read_text(encoding="utf-8")) + return [record for record in data.values() if not record.get("is_deprecated")] + + +def operator_page(prefix: str, record: dict) -> str: + """The docs/ path of an operator's page, as tools/update_di_reference.py writes it.""" + if record["pluginType"] == "transformer": + return f"{prefix}transformer/{record['main_category']}/{record['pluginId']}.md" + return f"{prefix}{record['pluginType']}/{record['pluginId']}.md" + + +def reference_overview(rule: SectionRule, md: str) -> str | None: + """The operator type whose overview page `md` is, in a `reference` section.""" + match = re.fullmatch(re.escape(rule.prefix) + r"([^/]+)/index\.md", md) + return match.group(1) if match else None + + +def reference_section( + entries: list[NavEntry], rule: SectionRule, operators: list[dict] +) -> tuple[list[NavEntry], set[str]]: + """Print the operators of a reference as compact entries after the overview page of their type (tasks/spec.md, §11). + + The section's own page is followed by a note on what an entry shows. Each + type's overview page is followed by its operators in alphabetical order; the + pages and navigation titles beneath it give way to them, so no page is + dropped. The operator pages in the navigation and the plugin descriptions + must agree. + """ + root = rule.prefix + "index.md" + catalog = {record["pluginId"]: (operator_page(rule.prefix, record), record["title"]) for record in operators} + pages = { + e.md for e in entries + if e.md and e.md.startswith(rule.prefix) and e.md != root and reference_overview(rule, e.md) is None + } + expected = {page for page, _ in catalog.values()} + if pages != expected: + without_page = sorted(plugin for plugin, (page, _) in catalog.items() if page not in pages) + raise click.ClickException( + f"{rule.prefix}: operator pages without a description in {OPERATORS_JSON}: " + f"{', '.join(sorted(pages - expected)) or 'none'}; descriptions without a page: " + f"{', '.join(without_page) or 'none'}" + ) + by_type: dict[str, list[dict]] = {} + for record in operators: + by_type.setdefault(record["pluginType"], []).append(record) + result: list[NavEntry] = [] + beneath: int | None = None + for entry in entries: + if beneath is not None: + if entry.depth > beneath: + continue + beneath = None + result.append(entry) + if entry.md == root: + result.append(NavEntry(entry.depth, generated=Generated("note", rule.prefix, "reference", [root]))) + kind = reference_overview(rule, entry.md) if entry.md else None + if kind is not None: + records = sorted(by_type.get(kind, []), key=lambda record: record["title"].casefold()) + result.append(NavEntry(entry.depth + 1, generated=Generated( + "operators", rule.prefix, "reference", [catalog[record["pluginId"]][0] for record in records], + operators=records, catalog=catalog, + ))) + beneath = entry.depth + return result, set() + + +def apply_section_rules( + entries: list[NavEntry], rules: list[SectionRule], operators: list[dict] | None = None +) -> tuple[list[NavEntry], set[str]]: + """Shorten the navigation by the section modes. Returns the entries and the pages dropped from them. + + A `reference` rule reads the plugin descriptions from data/plugins.json + unless `operators` passes them. + """ dropped: set[str] = set() for rule in rules: if not any(e.md and rule.matches(e.md) for e in entries): @@ -413,6 +519,10 @@ def apply_section_rules(entries: list[NavEntry], rules: list[SectionRule]) -> tu entries, gone = list_section(entries, rule) elif rule.mode == "omit": entries, gone = omit_section(entries, rule) + elif rule.mode == "reference": + if operators is None: + operators = load_operators() + entries, gone = reference_section(entries, rule, operators) else: continue dropped |= gone @@ -660,24 +770,228 @@ def section_url(generated: Generated, site_dir: Path, public_base: str) -> str: return urllib.parse.urljoin(public_base, path) -def render_generated(doc: BeautifulSoup, entry: NavEntry, site_dir: Path, public_base: str) -> list: +def render_generated( + doc: BeautifulSoup, entry: NavEntry, site_dir: Path, public_base: str, valid_ids: set | frozenset = frozenset() +) -> list: """The elements that stand in for a shortened section's pages.""" generated = entry.generated if generated.kind == "table": return [summary_table(doc, generated, site_dir)] if generated.kind == "groups": return [groups_table(doc, generated, site_dir)] + if generated.kind == "operators": + return operator_entries(doc, entry, site_dir, public_base, valid_ids) name = generated.name or page_title(site_dir, generated.pages[0]) url = section_url(generated, site_dir, public_base) note = doc.new_tag("div", attrs={"class": "admonition info"}) paragraph = doc.new_tag("p") - paragraph.string = ( - f"This print edition lists the {name} in short. The complete section is part of the online edition: {url}" - ) + if generated.mode == "reference": + paragraph.string = ( + f"This print edition lists every operator of the {name} with its description and parameters. " + f"The examples are part of the online edition: {url}. An operator marked Python plugin belongs to a " + "Python plugin package, which has to be installed first, for example with cmemc." + ) + else: + paragraph.string = ( + f"This print edition lists the {name} in short. The complete section is part of the online edition: {url}" + ) note.append(paragraph) return [note] +def data_type(parameter_type: str) -> str: + """How an entry names a parameter's data type (tasks/spec.md, §11).""" + if parameter_type.startswith("option[") and parameter_type.endswith("]"): + return f"{data_type(parameter_type[len('option['):-1])}, optional" + if parameter_type.startswith("code-"): + language = parameter_type[len("code-"):] + return CODE_LANGUAGES.get(language, language.upper()) + return DATA_TYPES.get(parameter_type, parameter_type) + + +def inline_markdown(text: str) -> list: + """A parameter description, rendered from its Markdown, as inline nodes.""" + if not text.strip(): + return [] + fragment = BeautifulSoup(markdown.markdown(text), "html.parser") + blocks = [node for node in fragment.contents if not (isinstance(node, NavigableString) and not node.strip())] + root = blocks[0] if len(blocks) == 1 and getattr(blocks[0], "name", None) == "p" else fragment + return [child.extract() for child in list(root.contents)] + + +def operator_fields(doc: BeautifulSoup, record: dict) -> Tag: + """An entry's field line: type, transformer category, plugin ID, Python plugin and a distance measure's range.""" + fields: list = [OPERATOR_TYPES.get(record["pluginType"], record["pluginType"])] + if record["pluginType"] == "transformer" and record.get("main_category"): + fields.append(record["main_category"]) + plugin_id = doc.new_tag("code") + plugin_id.string = record["pluginId"] + fields.append(plugin_id) + if record.get("backendType") == "python": + fields.append("Python plugin") + if record.get("distanceMeasureRange"): + fields.append(f"range: {record['distanceMeasureRange']}") + line = doc.new_tag("div", attrs={"class": "operator-fields"}) + for index, value in enumerate(fields): + if index: + line.append(" · ") + line.append(value) + return line + + +def operator_description(doc: BeautifulSoup, site_dir: Path, md: str) -> list: + """The description of an operator's page, as its entry prints it. + + Without the page title, the Python plugin note and the sections the entry + sets from the plugin description or leaves out; the page's own headings print + as run-in labels. + """ + article = page_article(site_dir, md) + if article is None: + return [] + title = article.find("h1") + if title is not None: + title.decompose() + for note in article.select("div.admonition"): + label = note.find(class_="admonition-title") + if label is not None and plain_heading(label).casefold() == "python plugin": + note.decompose() + for heading in article.find_all("h2"): + if heading.decomposed or plain_heading(heading).casefold() not in OPERATOR_SECTIONS_LEFT_OUT: + continue + for sibling in list(heading.find_next_siblings()): + if sibling.name == "h2": + break + sibling.decompose() + heading.decompose() + for heading in article.find_all(HEADING): + label = doc.new_tag("p") + strong = doc.new_tag("strong") + strong.string = plain_heading(heading) + label.append(strong) + heading.replace_with(label) + return [child.extract() for child in list(article.contents)] + + +def parameter_table(doc: BeautifulSoup, record: dict) -> list: + """An entry's parameter table, and the defaults too long for a cell (tasks/spec.md, §11). + + Parameter with its ID, Type, Default and Description; advanced parameters + after a row of their own, sub-parameters after their parameter. An operator + without parameters has no table. + """ + basic = list((record.get("properties") or {}).values()) + advanced = list((record.get("properties_advanced") or {}).values()) + if not basic and not advanced: + return [] + table, body = list_table(doc, ("Parameter", "Type", "Default", "Description")) + columns = doc.new_tag("colgroup") + for width in ("27%", "13%", "13%", "47%"): + columns.append(doc.new_tag("col", attrs={"style": f"width: {width}"})) + table.insert(0, columns) + blocks: list = [] + + def code(text: str) -> Tag: + element = doc.new_tag("code") + element.string = text + return element + + def add(parameter: dict, prefix: str = "") -> None: + ident = prefix + parameter["name"] + kind = parameter["parameterType"] + value = parameter.get("value") + name = list_cell(doc) + name.append(parameter["title"]) + name.append(doc.new_tag("br")) + name.append(code(ident)) + data = list_cell(doc) + data.string = data_type(kind) + default = list_cell(doc) + if kind in ("password", "objectParameter") or value in (None, "") or isinstance(value, dict): + default.string = "–" + elif "\n" in str(value): + default.string = "see below" + label = doc.new_tag("p") + label.append("Default of ") + label.append(code(ident)) + label.append(":") + block = doc.new_tag("pre") + content = code(str(value)) + if kind.startswith("code-"): + content["class"] = f"language-{kind[len('code-'):]}" + block.append(content) + blocks.extend([label, block]) + else: + default.append(code(str(value))) + description = list_cell(doc) + for node in inline_markdown(parameter.get("description") or ""): + description.append(node) + row = doc.new_tag("tr") + for cell in (name, data, default, description): + row.append(cell) + body.append(row) + for sub in (parameter.get("properties") or {}).values(): + add(sub, f"{ident}.") + + for parameter in basic: + add(parameter) + if advanced: + row = doc.new_tag("tr") + cell = list_cell(doc) + cell["colspan"] = "4" + strong = doc.new_tag("strong") + strong.string = "Advanced" + cell.append(strong) + row.append(cell) + body.append(row) + for parameter in advanced: + add(parameter) + return [table, *blocks] + + +def operator_related(doc: BeautifulSoup, record: dict, catalog: dict[str, tuple[str, str]]) -> Tag | None: + """The line naming an entry's related operators, each linked to its entry.""" + related = [catalog[ref["id"]] for ref in record.get("relatedPlugins") or [] if ref.get("id") in catalog] + if not related: + return None + line = doc.new_tag("p", attrs={"class": "operator-related"}) + line.append("Related: ") + for index, (md, title) in enumerate(related): + if index: + line.append(", ") + link = doc.new_tag("a", href=f"#{url_to_section_id(md_to_url_path(md))}") + link.string = title + line.append(link) + return line + + +def operator_entries( + doc: BeautifulSoup, entry: NavEntry, site_dir: Path, public_base: str, valid_ids: set | frozenset +) -> list: + """The compact entries of a reference's operators: heading, field line, description, parameters, related.""" + generated = entry.generated + level = min(6, entry.depth + 1) + sections = [] + for md, record in zip(generated.pages, generated.operators): + section_id = url_to_section_id(md_to_url_path(md)) + section = doc.new_tag("section", attrs={"class": "print-page", "id": section_id}) + for node in operator_description(doc, site_dir, md) + parameter_table(doc, record): + section.append(node) + namespace_ids(section, section_id) + resolve_links(section, md_to_url_path(md), valid_ids, public_base) + # Added after the ids and links are settled: namespacing would rewrite + # the related operators' in-document links. + heading = doc.new_tag(f"h{level}", attrs={"id": f"{section_id}-operator"}) + heading.string = record["title"] + section.insert(0, heading) + section.insert(1, operator_fields(doc, record)) + related = operator_related(doc, record, generated.catalog) + if related is not None: + section.append(related) + sections.append(section) + return sections + + def exclude_parts(doc: BeautifulSoup, article, page_url: str) -> int: """Replace the parts of a page marked `print-exclude` with a note naming the page online (tasks/spec.md, §10). @@ -724,17 +1038,29 @@ def merge_pages( """ doc = BeautifulSoup('', "html.parser") valid_ids = {url_to_section_id(md_to_url_path(e.md)) for e in entries if e.md} + # The operator pages of a reference print as entries, and links to them stay in the book. + valid_ids |= { + url_to_section_id(md_to_url_path(md)) + for e in entries if e.generated is not None and e.generated.kind == "operators" + for md in e.generated.pages + } dropped_ids = {url_to_section_id(md_to_url_path(md)) for md in dropped or ()} shortening = [rule for rule in rules or () if rule.mode != "full"] missing: list[str] = [] excluded: dict[str, int] = {} + part_count = 0 for entry in entries: if entry.generated is not None: - for element in render_generated(doc, entry, site_dir, public_base): + for element in render_generated(doc, entry, site_dir, public_base, valid_ids): doc.body.append(element) continue if entry.depth == 0: + # The print edition closes a part with the web addresses it cites + # (part-addresses in style.typ). + if print_edition and part_count: + doc.body.append(doc.new_tag("div", attrs={"class": "part-end"})) + part_count += 1 doc.body.append(doc.new_tag("div", attrs={"class": "chapter-break"})) if entry.md is None: heading = doc.new_tag(f"h{min(6, entry.depth + 1)}") @@ -755,6 +1081,11 @@ def merge_pages( parts = exclude_parts(doc, article, page_url) if parts: excluded[entry.md] = parts + if any(rule.mode == "reference" and reference_overview(rule, entry.md) for rule in rules or ()): + # The operator entries that follow replace the overview table. + for table in article.find_all("table"): + if not table.decomposed: + table.decompose() section_id = url_to_section_id(md_to_url_path(entry.md)) section = doc.new_tag("section", attrs={"class": "print-page", "id": section_id}) @@ -768,6 +1099,8 @@ def merge_pages( part_cover(doc, section) doc.body.append(section) + if print_edition and part_count: + doc.body.append(doc.new_tag("div", attrs={"class": "part-end"})) return doc, missing, excluded @@ -1191,7 +1524,13 @@ def build_pdf( rules = load_section_rules() if print_edition else [] entries, dropped = apply_section_rules(entries, rules) for rule in rules: - if rule.mode != "full": + if rule.mode == "reference": + count = sum( + len(e.generated.pages) for e in entries + if e.generated is not None and e.generated.kind == "operators" and e.generated.section == rule.prefix + ) + print(f"Print edition: {rule.prefix} as {rule.mode}, {count} operator entries") + elif rule.mode != "full": count = sum(1 for md in dropped if rule.matches(md)) print(f"Print edition: {rule.prefix} as {rule.mode}, {count} pages dropped") config = load_site_config() diff --git a/tools/pdf/authors.yml b/tools/pdf/authors.yml index fb39531c7..62b6499ca 100644 --- a/tools/pdf/authors.yml +++ b/tools/pdf/authors.yml @@ -1,57 +1,51 @@ --- -# Generated by `dec-tool pdf-authors` from the GitHub contributors of -# eccenca/documentation.eccenca.com and the names their profiles show. +# Generated by `dec-tool pdf-authors` from the commits to the pages of the print +# edition, their GitHub accounts in eccenca/documentation.eccenca.com and the names on their profiles. # Do not edit - run `task pdf:authors`. Names and exclusions: tools/pdf/print.yml. repository: eccenca/documentation.eccenca.com authors: - id: rpietzsch name: null - commits: 639 + commits: 249 - id: seebi name: Sebastian Tramp - commits: 532 - - id: irangareddy - name: Ranga Reddy - commits: 66 - - id: muddymudskipper - name: Thomas - commits: 62 - - id: mgns - name: Magnus - commits: 61 + commits: 156 - id: sobo name: Thomas Telleis - commits: 53 + commits: 46 + - id: muddymudskipper + name: Thomas + commits: 45 + - id: irangareddy + name: Ranga Reddy + commits: 37 - id: annamakor name: null - commits: 29 - - id: louiswesterheide - name: Louis - commits: 20 - - id: robertisele - name: Robert Isele - commits: 18 + commits: 22 - id: spl0tt name: Sebastian Plott - commits: 12 - - id: white-gecko - name: Natanael Arndt commits: 11 - - id: BorderCloud - name: Karima Rafes + - id: mgns + name: Magnus commits: 10 - - id: haschek - name: Michael Haschke + - id: white-gecko + name: Natanael Arndt commits: 10 + - id: louiswesterheide + name: Louis + commits: 9 + - id: robertisele + name: Robert Isele + commits: 9 - id: MaximilianWenzel name: null - commits: 9 - - id: tomatophantastico - name: Jörg Unbehauen - commits: 7 + commits: 6 - id: adelahaye-ecc name: null commits: 5 + - id: BorderCloud + name: Karima Rafes + commits: 1 - id: dgrtner-ecc name: null commits: 1 @@ -61,3 +55,6 @@ authors: - id: peterfreytag name: Peter Freytag commits: 1 + - id: tomatophantastico + name: Jörg Unbehauen + commits: 1 diff --git a/tools/pdf/filter.lua b/tools/pdf/filter.lua index 7d390af8d..44e3d2f11 100644 --- a/tools/pdf/filter.lua +++ b/tools/pdf/filter.lua @@ -41,6 +41,11 @@ function Div(el) return raw('#chapter-break()') elseif c:includes('part-contents') then return raw('#part-contents()') + elseif c:includes('operator-fields') then + return wrap('#operator-fields[', el.content, ']') + elseif c:includes('part-end') then + -- The web addresses the part cites; the print edition only. + return raw('#part-addresses()') elseif c:includes('admonition') then local kind = 'note' for _, cl in ipairs(c) do diff --git a/tools/pdf/print.yml b/tools/pdf/print.yml index 80955db36..002eaaa0f 100644 --- a/tools/pdf/print.yml +++ b/tools/pdf/print.yml @@ -61,10 +61,13 @@ body-weight: light # one table of the titles and their page names # omit - left out without a trace; the imprint names the online edition as # the complete reference. The only mode a page accepts +# reference - the operator reference only: each type's overview page keeps +# its introduction, and its operators follow as compact entries from +# data/plugins.json instead of the overview table (tasks/spec.md, §11) # A part of a page is left out with the class `print-exclude` in the Markdown, # e.g. ```sql { .print-exclude }; a note naming the page online takes its place. sections: - build/reference/: list + build/reference/: reference release-notes/: mode: list columns: [Release, Summary] diff --git a/tools/pdf/style.typ b/tools/pdf/style.typ index 9737b831b..cea02403a 100644 --- a/tools/pdf/style.typ +++ b/tools/pdf/style.typ @@ -159,16 +159,50 @@ } }) -#let cards(..items) = grid(columns: (1fr, 1fr), column-gutter: 10pt, row-gutter: 10pt, ..items.pos()) - // A card is not breakable: split across pages, its title and rule would stay // behind on one page and its text start the next. -#let card(body) = block(width: 100%, breakable: false, stroke: 0.5pt + ec-grey.lighten(55%), radius: 2pt, inset: 10pt, { - set text(size: size-small) - set par(spacing: 1em, justify: false) - set block(spacing: 0.9em) - set list(indent: 0.2em, body-indent: 0.8em) - body +#let card-frame(body, height: auto) = block( + width: 100%, + height: height, + breakable: false, + stroke: 0.5pt + ec-grey.lighten(55%), + radius: 2pt, + inset: 10pt, + { + set text(size: size-small) + set par(spacing: 1em, justify: false) + set block(spacing: 0.9em) + set list(indent: 0.2em, body-indent: 0.8em) + body + }, +) + +// tools/pdf/filter.lua passes each card's content through card() to cards(). +#let card(body) = body + +// The field line under an operator's title in the compact operator reference: +// type, category, plugin ID (tasks/spec.md, §11). Kept with what follows. +#let operator-fields(body) = block(above: 0.35em, below: 0.9em, sticky: true, { + set par(justify: false) + text(size: size-small, fill: ec-grey, body) +}) + +// DEVIATION: facing cards end on one line (tasks/spec.md, §11). The cards of a +// row are measured at the column width and both framed at the height of the +// taller one; a card alone in the last row keeps its own height. +#let cards(..items) = layout(region => { + let bodies = items.pos() + let gutter = 10pt + let width = (region.width - gutter) / 2 + let cells = () + for row in range(0, bodies.len(), step: 2) { + let pair = bodies.slice(row, calc.min(row + 2, bodies.len())) + let height = if pair.len() < 2 { auto } else { + calc.max(..pair.map(body => measure(card-frame(body), width: width).height)) + } + cells += pair.map(body => card-frame(body, height: height)) + } + grid(columns: (1fr, 1fr), column-gutter: gutter, row-gutter: gutter, ..cells) }) // DEVIATION: a policy links nowhere outside itself. The docs link to other @@ -223,7 +257,10 @@ let pages = query(heading.where(level: 2).after(parts.last().location())).filter(started) if pages.len() > 0 { pages.last() } else { parts.last() } } - let title = if running == none { [] } else { numbered-title(running) } + // A left-hand page names its part as the part's title band does: Part A: Build. + let title = if running == none { [] } else if calc.even(page-no) and running.level == 1 { + [Part #numbering("A", ..counter(heading).at(running.location())): #running.body] + } else { numbered-title(running) } if calc.even(page-no) { grid(columns: (auto, 1fr), column-gutter: 1.2em, folio, title) } else { @@ -263,6 +300,65 @@ chapter-break() } +// DEVIATION (print): a link out of the book carries a superscript number, and +// its part ends with a list of the web addresses it cites (tasks/spec.md, §11, +// D15). The numbers run within a part, and an address cited again keeps its +// number. The list is set as text, without link annotations - Typst footnotes +// link marker and entry, and PDF/X output wants no annotations. +#let part-start() = query(heading.where(level: 1).before(here())).at(-1, default: none) + +#let cited-addresses() = { + let part = part-start() + let cited = selector() + query(if part == none { cited } else { cited.after(part.location()) }.before(here())) +} + +#let web-address(address) = { + [#metadata(address) ] + context { + let order = () + for cited in cited-addresses() { + if cited.value not in order { order.push(cited.value) } + } + super(str(order.position(value => value == address) + 1)) + } +} + +// The build places the call at the end of every part (tools/build_pdf.py). +#let part-addresses() = if print-edition { + context { + let order = () + let pages = (:) + for cited in cited-addresses() { + let page = numbering("1", ..counter(page).at(cited.location())) + let known = pages.at(cited.value, default: none) + if known == none { + order.push(cited.value) + pages.insert(cited.value, (page,)) + } else if page not in known { + pages.insert(cited.value, known + (page,)) + } + } + if order.len() > 0 { + pagebreak(weak: true) + heading(level: 2, numbering: none, outlined: false, bookmarked: true)[Web addresses] + set text(size: size-table) + set par(justify: false) + show regex("[/._-]"): mark => mark + sym.zws + grid( + columns: (auto, 1fr), + column-gutter: 0.8em, + row-gutter: 0.65em, + ..order.enumerate().map(((index, address)) => { + let cited = pages.at(address) + let label = if cited.len() == 1 { "p." } else { "pp." } + (str(index + 1), [#address #text(fill: ec-grey)[(#label~#cited.join(", "))]]) + }).flatten(), + ) + } + } +} + // DEVIATION (print): the back of the title page is the imprint - edition, // publisher, authors, licence and the online edition - and it carries the // edition stamp the screen PDF prints in its header. tools/pdf/print.yml holds @@ -278,7 +374,7 @@ v(1fr) entry(title)[#context-line, #subtitle \ Print edition, generated #generated] entry("Publisher")[#config.publisher.name \ #config.publisher.address.join(linebreak())] - entry("Authors")[The contributors to the documentation, most commits first: #authors.] + entry("Authors")[The contributors to the documentation, most commits to the printed pages first: #authors.] // The URL is a string: written as markup, Typst would turn it into a link. entry("Licence")[ This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License, @@ -415,8 +511,9 @@ set enum(indent: 1.84em, body-indent: 1.64em, spacing: 0.85em) set block(spacing: if print-edition { 1.0em } else { 1.52em }) // DEVIATION (print): paper cannot follow a link. A link within the book prints - // the page it leads to, a link out of the book its address in a footnote, and - // neither leaves a link in the PDF (tasks/spec.md, R4). + // the page it leads to, a link out of the book a number whose address its part + // lists at the end (part-addresses), and neither leaves a link in the PDF + // (tasks/spec.md, R4, D15). show link: it => if not print-edition { text(fill: ec-link, underline(it)) if type(it.dest) == str and it.dest.starts-with(regex("https?://")) { external-mark } @@ -429,8 +526,8 @@ } else if type(it.dest) == str { let address = it.dest.trim("mailto:", at: start) it.body - // An address that prints as its own text needs no footnote repeating it. - if it.body.at("text", default: none) != address { footnote(address) } + // An address that prints as its own text needs no number repeating it. + if it.body.at("text", default: none) != address { web-address(address) } } else { it.body } diff --git a/tools/pdf_authors.py b/tools/pdf_authors.py index 176fd9e23..3c4bda1d1 100644 --- a/tools/pdf_authors.py +++ b/tools/pdf_authors.py @@ -1,10 +1,11 @@ """Write the author list the imprint of the print edition prints. -The authors are the GitHub accounts that contributed to the documentation -repository, most commits first, each with the name its GitHub profile shows. -Anonymous contributions - commits whose e-mail maps to no GitHub account - bot -accounts, software agents and the IDs excluded in tools/pdf/print.yml are not -listed, and the names of excluded IDs are never looked up. +The authors are the GitHub accounts behind the commits to what the print +edition prints - its pages that are not generated, and the images they +reference - most commits first, each with the name its GitHub profile shows. +Anonymous commits - whose e-mail maps to no GitHub account - bot accounts, +software agents and the IDs excluded in tools/pdf/print.yml are not listed, and +the names of excluded IDs are never looked up. The list is committed as tools/pdf/authors.yml, so a PDF build stays offline and reproducible, and a changed list shows up in review. The hand-maintained @@ -14,7 +15,8 @@ does not list yet, without a name, so every name to maintain is in one place (`prefill_names`). Invoked by `task pdf:authors`. Requests use GITHUB_TOKEN or GH_TOKEN, else the token of a logged-in GitHub CLI: without a token GitHub -allows 60 requests an hour, and a run takes one per author. +allows 60 requests an hour, and a run takes one per author and one per hundred +commits. """ from __future__ import annotations @@ -28,6 +30,8 @@ import urllib.error import urllib.parse import urllib.request +from collections import Counter +from concurrent.futures import ThreadPoolExecutor from dataclasses import dataclass, field from pathlib import Path from typing import Callable @@ -262,17 +266,107 @@ def get_json(url: str, token: str | None) -> tuple[object, str | None]: raise click.ClickException(f"cannot reach the GitHub API ({getattr(error, 'reason', error)})") from error -def fetch_contributors(repository: str, token: str | None) -> list[dict]: - """Every contributor of a repository, following the API's pagination. +# The marker a generated page carries (.claude/docs-guidelines/repo-conventions.md). +GENERATED_MARK = re.compile(r"generated - do not change it manually|auto-generated", re.IGNORECASE) +# An image a page references, in Markdown or as an element. +IMAGE_REFERENCE = re.compile(r'!\[[^\]]*\]\(\s*]+)|]*\bsrc="([^"]+)"') - Without `anon=1` the API leaves out contributions that map to no account. + +def printed_pages() -> list[str]: + """The pages the print edition prints, after the section rules of print.yml.""" + # Imported here: tools.build_pdf imports this module for the imprint. + from tools.build_pdf import apply_section_rules, load_nav_entries, load_section_rules + + entries, _ = apply_section_rules(load_nav_entries(), load_section_rules()) + return [entry.md for entry in entries if entry.md] + + +def printed_files(pages: list[str], docs_dir: Path = Path("docs")) -> list[Path]: + """The source files of printed pages: the pages that are not generated, and the images they reference. + + A generated page credits whoever ran its generator, so it does not count. """ - url: str | None = f"{API}/repos/{repository}/contributors?per_page=100" - contributors: list[dict] = [] + files: list[Path] = [] + for md in pages: + page = docs_dir / md + if not page.is_file(): + continue + text = page.read_text(encoding="utf-8", errors="ignore") + if GENERATED_MARK.search("\n".join(text.splitlines()[:15])): + continue + files.append(page) + for match in IMAGE_REFERENCE.finditer(text): + source = urllib.parse.unquote((match.group(1) or match.group(2)).split("#")[0].split("?")[0]) + if not source or source.startswith(("http://", "https://", "data:", "/")): + continue + image = Path(os.path.normpath(page.parent / source)) + if image.is_file() and image not in files: + files.append(image) + return files + + +def file_commits(files: list[Path], repo: Path = Path("."), env: dict | None = None) -> dict[str, str]: + """The commits of the files, without merges and following renames: each hash with its author's e-mail.""" + + def log(path: Path) -> list[list[str]]: + result = subprocess.run( + ["git", "log", "--no-merges", "--follow", "--format=%H%x09%ae", "--", str(path)], + cwd=repo, env=env, capture_output=True, text=True, check=False, + ) + return [line.split("\t", 1) for line in result.stdout.splitlines() if "\t" in line] + + commits: dict[str, str] = {} + # One `git log` per file: --follow follows a single path only. + with ThreadPoolExecutor(max_workers=8) as pool: + for rows in pool.map(log, files): + commits.update(rows) + return commits + + +def commit_accounts(commits: list[dict]) -> tuple[dict[str, dict], dict[str, dict]]: + """The GitHub accounts behind commits, from the commits API: by commit hash, and by author e-mail. + + The e-mail names the account of a commit the API does not list - one that is + not pushed yet, or that was made on another branch. + """ + by_sha: dict[str, dict] = {} + by_email: dict[str, dict] = {} + for item in commits: + author = item.get("author") or {} + if not author.get("login"): + continue + account = {"login": author["login"], "type": author.get("type", "User")} + by_sha[item["sha"]] = account + email = ((item.get("commit") or {}).get("author") or {}).get("email") + if email: + by_email.setdefault(email.casefold(), account) + return by_sha, by_email + + +def count_commits(commits: dict[str, str], by_sha: dict[str, dict], by_email: dict[str, dict]) -> list[dict]: + """Contributor records - login, type and commits - counting each commit once. + + A commit that maps to no account is anonymous and left out. + """ + counts: Counter = Counter() + types: dict[str, str] = {} + for sha, email in commits.items(): + account = by_sha.get(sha) or by_email.get(email.casefold()) + if account is None: + continue + counts[account["login"]] += 1 + types[account["login"]] = account["type"] + return [{"login": login, "type": types[login], "contributions": count} for login, count in counts.items()] + + +def fetch_commits(repository: str, token: str | None) -> list[dict]: + """Every commit the GitHub API lists for the repository's default branch, following the pagination.""" + url: str | None = f"{API}/repos/{repository}/commits?per_page=100" + commits: list[dict] = [] while url: page, url = get_json(url, token) - contributors.extend(page) - return contributors + commits.extend(page) + return commits def fetch_profile_name(login: str, token: str | None) -> str | None: @@ -292,8 +386,8 @@ def authors_yaml(repository: str, authors: list[dict]) -> str: """The committed file: a header naming its source, then the list in imprint order.""" header = ( "---\n" - "# Generated by `dec-tool pdf-authors` from the GitHub contributors of\n" - f"# {repository} and the names their profiles show.\n" + "# Generated by `dec-tool pdf-authors` from the commits to the pages of the print\n" + f"# edition, their GitHub accounts in {repository} and the names on their profiles.\n" "# Do not edit - run `task pdf:authors`. Names and exclusions: tools/pdf/print.yml.\n" ) body = yaml.dump( @@ -307,7 +401,7 @@ def authors_yaml(repository: str, authors: list[dict]) -> str: @click.option( "--repository", default=REPOSITORY, - help="Which GitHub repository's contributors are the authors?", + help="Which GitHub repository maps the commits to accounts?", show_default=True, ) @click.option( @@ -318,7 +412,7 @@ def authors_yaml(repository: str, authors: list[dict]) -> str: show_default=True, ) def pdf_authors(repository: str, output_file: str) -> None: - """Write the imprint's author list from the repository's GitHub contributors and their profile names.""" + """Write the imprint's author list from the commits to the printed pages and the authors' profile names.""" token, source = github_token() if source: # Flushed, so the line comes before an error message also when the output is piped. @@ -326,13 +420,17 @@ def pdf_authors(repository: str, output_file: str) -> None: else: print( "WARNING: no GitHub token - GitHub allows 60 requests an hour without one, and this run takes one " - f"per author; {TOKEN_HINT}", + f"per author and one per hundred commits; {TOKEN_HINT}", file=sys.stderr, ) rules = load_author_rules() - authors = select_authors(fetch_contributors(repository, token), rules) + files = printed_files(printed_pages()) + commits = file_commits(files) + print(f"{len(commits)} commits to {len(files)} printed files", flush=True) + by_sha, by_email = commit_accounts(fetch_commits(repository, token)) + authors = select_authors(count_commits(commits, by_sha, by_email), rules) if not authors: - raise click.ClickException(f"no contributors found for {repository}") + raise click.ClickException("no author found in the commits to the printed pages") authors = add_names(authors, lambda login: fetch_profile_name(login, token)) Path(output_file).write_text(authors_yaml(repository, authors), encoding="utf-8") print(f"{len(authors)} authors written to {output_file}") From f4da226caaaf1f4a4e67c1f087e7d11e0a049ee3 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Wed, 16 Sep 2026 07:38:14 +0200 Subject: [PATCH 06/15] add PDF/X-4 normalization and a preflight report for the print edition Ghostscript 10.08 crashes on the colour emoji Typst writes as a Type 3 font, and silently drops transparent SVG content when it converts to PDF/X-4, so the print edition renders both to images first. - dec-tool pdf-normalize writes the book block as PDF/X-4 in CMYK, or a greyscale preview with --gray; the ISO Coated v2 profile is fetched into dist/icc/ rather than vendored, as its licence forbids that - dec-tool pdf-preflight checks page size and count, fonts, image resolution, transparency, annotations, page numbers and grey areas; the build runs it before and after the conversion - dec-tool image-widths writes a screenshot's missing width into the Markdown, from its pixels and capture density: 24 widths in 14 pages - the print build lists images below 150 ppi in low-resolution.tsv, minus the ones accepted in print.yml - a short lead line under a heading stays with it, so no heading ends a page alone --- .claude/docs-guidelines/repo-conventions.md | 4 + Taskfile.yml | 13 +- .../index.md | 2 +- .../index.md | 2 +- docs/build/rule-blocks/index.md | 4 +- docs/build/rule-operators/index.md | 2 +- .../define-the-interfaces/index.md | 6 +- .../index.md | 8 +- .../index.md | 4 +- .../link-IDS-event-to-KG/index.md | 2 +- docs/build/variables/index.md | 2 +- docs/build/workflows/index.md | 2 +- docs/distribution/marketplace/index.md | 8 +- .../sparql-constraints/index.md | 2 +- .../workflow-trigger/index.md | 2 +- .../workspace-configuration/index.md | 2 +- tasks/backlog.md | 83 +++- tasks/spec.md | 53 ++- tests/test_build_pdf_emoji.py | 110 +++++ tests/test_build_pdf_print.py | 22 + tests/test_build_pdf_resolution.py | 60 +++ tests/test_image_widths.py | 80 ++++ tests/test_pdf_normalize.py | 192 ++++++++ tests/test_pdf_preflight.py | 181 ++++++++ tools/README.md | 3 + tools/__init__.py | 6 + tools/build_pdf.py | 320 ++++++++++++- tools/image_widths.py | 133 ++++++ tools/pdf/filter.lua | 4 + tools/pdf/print.yml | 5 + tools/pdf/style.typ | 17 + tools/pdf_normalize.py | 317 +++++++++++++ tools/pdf_preflight.py | 425 ++++++++++++++++++ 33 files changed, 2029 insertions(+), 47 deletions(-) create mode 100644 tests/test_build_pdf_emoji.py create mode 100644 tests/test_build_pdf_resolution.py create mode 100644 tests/test_image_widths.py create mode 100644 tests/test_pdf_normalize.py create mode 100644 tests/test_pdf_preflight.py create mode 100644 tools/image_widths.py create mode 100644 tools/pdf_normalize.py create mode 100644 tools/pdf_preflight.py diff --git a/.claude/docs-guidelines/repo-conventions.md b/.claude/docs-guidelines/repo-conventions.md index f23ca681b..bbda5f6c3 100644 --- a/.claude/docs-guidelines/repo-conventions.md +++ b/.claude/docs-guidelines/repo-conventions.md @@ -127,6 +127,10 @@ Do not write `index.html` stubs under `docs/`. - `class="bordered"` on every product screenshot (436 uses in the tree). - `width="50%"` or `width="70%"` for dialogs and modals, no width for full-screen views. Keep the width consistent within one page. +- A screenshot narrower than the page needs a width, or the print edition prints it too small or too + coarse. `poetry run dec-tool image-widths` lists raster images without one in pages that are not + generated, and `--fix` writes `width="NN%"` from the pixel width and the capture density (a Retina + capture at 144 dpi counts half), against the 16 cm print column. An image as wide as the page gets none. - `.off-glb` opts an image out of the glightbox lightbox — used for inline icons and decorative images. - File names: lowercase, hyphen-separated, descriptive (style guide 5.5). - Delete screenshots that are no longer referenced. Orphan check for one page directory: diff --git a/Taskfile.yml b/Taskfile.yml index 64fea8820..fc24bed24 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -218,12 +218,23 @@ tasks: Writes dist/documentation-eccenca-com--print.pdf; the merged HTML, the Typst source and the normalized images stay in dist/pdf/print/. The screen PDF of `task pdf` is not affected. + + The build lists images below 150 ppi at their printed size in + dist/pdf/print/low-resolution.tsv and runs the preflight report + (`dec-tool pdf-preflight`), failing on an error. `task pdf:print -- + --normalize` (or PDF_NORMALIZE=1) also writes a PDF/X-4 copy in CMYK + with Ghostscript, -print-x4.pdf, and checks that one; the ISO Coated v2 + profile is fetched from the ECI into dist/icc/ unless PDF_ICC_PROFILE + names one. `task pdf:print -- --gray` (or PDF_GRAY=1) writes a + greyscale preview, -print-gray-x4.pdf, to check on screen how the book + prints in black and white; it is PDF/X-4 by the grey profile Ghostscript + ships, and the two flags combine. deps: - task: build env: BUILD_VERSION: '{{.CURRENT_VERSION}}' cmds: - - poetry run dec-tool build-pdf --edition print + - poetry run dec-tool build-pdf --edition print {{.CLI_ARGS}} pdf:authors: desc: Refresh the author list of the print edition's imprint diff --git a/docs/automate/processing-data-with-variable-input-workflows/index.md b/docs/automate/processing-data-with-variable-input-workflows/index.md index 1470475dc..169067499 100644 --- a/docs/automate/processing-data-with-variable-input-workflows/index.md +++ b/docs/automate/processing-data-with-variable-input-workflows/index.md @@ -42,7 +42,7 @@ Click the :eccenca-module-marketplace: **Packages** icon in the main menu under Search for the required ontologies / vocabularies and click the **Install** button. Wait for a package installation to complete (the **Install** button will change to **Uninstall**) before installing the next package. -![Search and install ontologies](pdwviw-marketplace-schema-search.png){ class="bordered" } +![Search and install ontologies](pdwviw-marketplace-schema-search.png){ class="bordered" width="90%" } ## 2 Create a new project diff --git a/docs/build/loading-jdbc-datasets-incrementally/index.md b/docs/build/loading-jdbc-datasets-incrementally/index.md index 3461908b3..142a0c893 100644 --- a/docs/build/loading-jdbc-datasets-incrementally/index.md +++ b/docs/build/loading-jdbc-datasets-incrementally/index.md @@ -37,7 +37,7 @@ To extract data from a relational database, you need to first register a **JDBC 1. In Corporate Memory, click Projects under **BUILD** in the navigation on the left side of the page. - ![Menu BUILD > Projects](incr-jdbc-menu-build-projects.png){ class="bordered" } + ![Menu BUILD > Projects](incr-jdbc-menu-build-projects.png){ class="bordered" width="45%" } 2. Click **Create** at the top of the page. 3. In the Create new item window, select Project and click Add. The Create new item of type Project window appears. diff --git a/docs/build/rule-blocks/index.md b/docs/build/rule-blocks/index.md index 6be5611ba..8efb54119 100644 --- a/docs/build/rule-blocks/index.md +++ b/docs/build/rule-blocks/index.md @@ -89,7 +89,7 @@ Dragging **New input port** onto the canvas creates a port as well. Existing ports are listed below that entry. Click the **:material-pencil-outline: edit** icon of a port to open the **Edit input port** dialog, or the **:material-delete-outline: delete** icon to remove it: -![The Edit input port dialog](rule-blocks-input-port.png "The Edit input port dialog") +![The Edit input port dialog](rule-blocks-input-port.png "The Edit input port dialog"){ width="95%" } **Label** : The name of the port. @@ -145,7 +145,7 @@ It has one input handle per input port, in the port order defined in the rule bl Its context menu offers the actions specific to the reference: -![The context menu of a rule block node](rule-blocks-node-menu.png "The context menu of a rule block node") +![The context menu of a rule block node](rule-blocks-node-menu.png "The context menu of a rule block node"){ width="50%" } **Clone** : Adds a second reference to the same rule block. diff --git a/docs/build/rule-operators/index.md b/docs/build/rule-operators/index.md index e7a94e531..815172779 100644 --- a/docs/build/rule-operators/index.md +++ b/docs/build/rule-operators/index.md @@ -54,7 +54,7 @@ The following parameters can be set for each comparison: !!! info inline end "" - ![Confidence graph](confidence-graph.png) + ![Confidence graph](confidence-graph.png){ width="60%" } The threshold is used to convert the computed distance to a confidence between -1.0 and 1.0. Links will be generated for confidences above 0 while higher confidence values imply a higher similarity between the compared entities. diff --git a/docs/build/tutorial-how-to-link-ids-to-osint/define-the-interfaces/index.md b/docs/build/tutorial-how-to-link-ids-to-osint/define-the-interfaces/index.md index 23fe2db7c..61e332c7f 100644 --- a/docs/build/tutorial-how-to-link-ids-to-osint/define-the-interfaces/index.md +++ b/docs/build/tutorial-how-to-link-ids-to-osint/define-the-interfaces/index.md @@ -10,21 +10,21 @@ In this tutorial, we will show how to (1) define the information available in an In the previous tutorial, we have defined the use cases in contact with the humans, ie. analysts (see figure 1)
-![Figure 1. We need to imagine an interface where analysts can list the IoCs during the incident and read all their documentations.](use_cases_with_interfaces.png) +![Figure 1. We need to imagine an interface where analysts can list the IoCs during the incident and read all their documentations.](use_cases_with_interfaces.png){ width="50%" }
Figure 1. We need to imagine an interface where analysts can list the IoCs during the incident and read all their documentations.
The classic Splunk interface is a set of panels, like "static table" panel. This table panel can show a table of cells and also one cell with a text via in input a Splunk Search Processing Language (SPL). With the plugin [Linked Data App](../link-IDS-event-to-KG/eccenca_commands.tar.gz) ([tutorial page](../link-IDS-event-to-KG/index.md)), we can insert a SPARQL query and select the part of your knowledge graph to print (figure 2).
-![Figure 2. An analyst can insert a SPARQL query with Splunk token in input of one "static table" panel of his dashboard with the plugin "Linked Data App"](demo-splunk-with-kg-edit.png) +![Figure 2. An analyst can insert a SPARQL query with Splunk token in input of one "static table" panel of his dashboard with the plugin "Linked Data App"](demo-splunk-with-kg-edit.png){ width="75%" }
Figure 2. An analyst can insert a SPARQL query with Splunk token in input of one "static table" panel of his dashboard with the plugin "Linked Data App"
The first dashboard to do for our use cases is the list of IoCs with classic SPL queries of analysts via a static table and allow the analyst to select one IoC via a click in the table. The dashboard with this selected row can save the ID of IoC in a global variable for the other panels in the same dashboard ([a Splunk token](https://docs.splunk.com/Documentation/Splunk/9.0.5/Viz/tokens)). When this variable (Splunk token) is changed by the user, Splunk is able to recalculate automatically the queries with this variable in the other static tables. So with this mechanism, we can print the details in the knowledge graph (with SPARQL queries) and the IoC statistics in the Splunk indexes (with SPL queries) around of one selected IoC. With these knowledge about Splunk dashboard, we proposed to analysts a first naive interface in the figure 3.
-![Figure 3. Imagine the expected Splunk dashboard with its interaction](interface.png) +![Figure 3. Imagine the expected Splunk dashboard with its interaction](interface.png){ width="90%" }
Figure 3. Imagine the expected Splunk dashboard with its interaction
diff --git a/docs/build/tutorial-how-to-link-ids-to-osint/lift-data-from-STIX-2.1-data-of-mitre-attack/index.md b/docs/build/tutorial-how-to-link-ids-to-osint/lift-data-from-STIX-2.1-data-of-mitre-attack/index.md index 8d176897b..22e50d0df 100644 --- a/docs/build/tutorial-how-to-link-ids-to-osint/lift-data-from-STIX-2.1-data-of-mitre-attack/index.md +++ b/docs/build/tutorial-how-to-link-ids-to-osint/lift-data-from-STIX-2.1-data-of-mitre-attack/index.md @@ -77,7 +77,7 @@ Each domain dataset is saved in GitHub: !!! Success Now, you can see these JSON datasets in Corporate Memory: - ![](23-1-import-JSON-result.png) + ![](23-1-import-JSON-result.png){ width="80%" } ### Create named graphs @@ -118,7 +118,7 @@ Create one RDF dataset for each Mitre dataset: !!! Success Now, you can see these RDF datasets in Corporate Memory: - ![](23-1-create-RDF-dataset-result.png) + ![](23-1-create-RDF-dataset-result.png){ width="80%" } !!! Tip @@ -404,7 +404,7 @@ Create one RDF dataset for each Mitre dataset: You can test the result when you search the Mitre ID via the explorer of knowledge graph "MA Entreprise": - ![](23-1-success-extract-rdfslabel.png) + ![](23-1-success-extract-rdfslabel.png){ width="55%" } ### Create a workflow @@ -754,7 +754,7 @@ After this tutorial, you want probably to navigate in your new knowledge graph b 5. Create a new workflow "MITRE ATT&CK® workflow" where you will insert the other workflows, like that: - ![](23-1-ex-workflow-global.png) + ![](23-1-ex-workflow-global.png){ width="75%" } !!! Success diff --git a/docs/build/tutorial-how-to-link-ids-to-osint/lift-data-from-YAML-data-of-hayabusa-sigma/index.md b/docs/build/tutorial-how-to-link-ids-to-osint/lift-data-from-YAML-data-of-hayabusa-sigma/index.md index ec1c1fa19..08db384d9 100644 --- a/docs/build/tutorial-how-to-link-ids-to-osint/lift-data-from-YAML-data-of-hayabusa-sigma/index.md +++ b/docs/build/tutorial-how-to-link-ids-to-osint/lift-data-from-YAML-data-of-hayabusa-sigma/index.md @@ -134,7 +134,7 @@ This new transformer are building the following RDF model for your use case: - replace `\./hayabusa-rules/` by `https://github.com/Yamato-Security/hayabusa-rules/blob/main/` - replace `\./sigma/` by `https://github.com/SigmaHQ/sigma/blob/master/` - ![](23-1-rules-isdefinedby.png) + ![](23-1-rules-isdefinedby.png){ width="95%" } So the rulepath `./sigma/rules/windows/process_creation/proc_creation_win_bcdedit_boot_conf_tamper.yml` becomes the link `https://github.com/SigmaHQ/sigma/blob/master/rules/windows/process_creation/proc_creation_win_bcdedit_boot_conf_tamper.yml` and `./hayabusa-rules/hayabusa/sysmon/Sysmon_15_Info_ADS-Created.yml`becomes `https://github.com/Yamato-Security/hayabusa-rules/blob/main/hayabusa/sysmon/Sysmon_11_Med_FileCreated_RuleAlert.yml` @@ -155,7 +155,7 @@ This new transformer are building the following RDF model for your use case: And don't forget to allow the replacement of JSON dataset because it allows to replace this specific JSON by all other rules during the execution of this worflow. - ![](23-1-workflow-allow-replacement.png) + ![](23-1-workflow-allow-replacement.png){ width="65%" } ![](23-1-add-worflow.gif) diff --git a/docs/build/tutorial-how-to-link-ids-to-osint/link-IDS-event-to-KG/index.md b/docs/build/tutorial-how-to-link-ids-to-osint/link-IDS-event-to-KG/index.md index 776a74185..3e4fa775d 100644 --- a/docs/build/tutorial-how-to-link-ids-to-osint/link-IDS-event-to-KG/index.md +++ b/docs/build/tutorial-how-to-link-ids-to-osint/link-IDS-event-to-KG/index.md @@ -27,7 +27,7 @@ The "Linked Data App" extends Splunk Search Processing Language (SPL) to support 2. Open the App window in Splunk via the icon "tools" (see figure 1)
-![Figure 1: In the top of the list of installed Splunk apps, you need to click on the icon "tools" to open the window to manage your apps](splunk_apps_menu.png) +![Figure 1: In the top of the list of installed Splunk apps, you need to click on the icon "tools" to open the window to manage your apps](splunk_apps_menu.png){ width="30%" }
Figure 1: In the top of the list of installed Splunk apps, you need to click on the icon "tools" to open the window to manage your apps
diff --git a/docs/build/variables/index.md b/docs/build/variables/index.md index a55d06ce0..1257862b4 100644 --- a/docs/build/variables/index.md +++ b/docs/build/variables/index.md @@ -225,7 +225,7 @@ Select the `project.email_ids` and include the double curly brackets at last. Same as above, add the variable click on :eccenca-item-add-artefact: and type name as `password`, value as `xmhnjnnjkmnlbbhbvfhnbjkm` and in the description `Gmail account password` as shown below then click on **Add**. -![](di-var-password-definition.png){ class="bordered" } +![](di-var-password-definition.png){ class="bordered" width="95%" } !!! success "Step Result" diff --git a/docs/build/workflows/index.md b/docs/build/workflows/index.md index c4c8ca338..30fc40b19 100644 --- a/docs/build/workflows/index.md +++ b/docs/build/workflows/index.md @@ -55,7 +55,7 @@ The **Clear Dataset** operator empties the dataset connected to its output befor Place the Clear Dataset operator in the workflow and connect its output to the dataset that should be cleared. The operator takes no data inputs; connect it using a **dependency connection** from the upstream node that must complete first, or leave it unconnected if the dataset should be cleared before any subsequent nodes execute. -![Clear Datasets](wf-clear-datasets.png) +![Clear Datasets](wf-clear-datasets.png){ width="90%" } Some datasets historically provided their own clear attributes (e.g., `Clear graph before workflow execution` on the Knowledge Graph dataset). These per-dataset attributes are **deprecated** and should no longer be used. Use the Clear Dataset operator instead, which works uniformly across all dataset types. diff --git a/docs/distribution/marketplace/index.md b/docs/distribution/marketplace/index.md index af7fb3888..3865bc5b3 100644 --- a/docs/distribution/marketplace/index.md +++ b/docs/distribution/marketplace/index.md @@ -61,7 +61,7 @@ They can be combined. !!! info inline "" - ![The open Package type drop-down](marketplace-filter-package-type.png "The open Package type drop-down") + ![The open Package type drop-down](marketplace-filter-package-type.png "The open Package type drop-down"){ width="35%" } #### Package type @@ -73,7 +73,7 @@ Show only **Vocabulary** or only **Project** packages, see [Package Types](#pack !!! info inline "" - ![The open License drop-down](marketplace-filter-license.png "The open License drop-down") + ![The open License drop-down](marketplace-filter-license.png "The open License drop-down"){ width="50%" } #### License @@ -93,7 +93,7 @@ Show only packages whose name or description contains the entered keyword. !!! info inline "" - ![The Installed toggle, switched on](marketplace-filter-installed.png "The Installed toggle, switched on") + ![The Installed toggle, switched on](marketplace-filter-installed.png "The Installed toggle, switched on"){ width="20%" } #### Installed @@ -105,7 +105,7 @@ Switch on to show only the packages currently installed in your instance. !!! info inline "" - ![The open Marketplace drop-down](marketplace-filter-server.png "The open Marketplace drop-down") + ![The open Marketplace drop-down](marketplace-filter-server.png "The open Marketplace drop-down"){ width="60%" } #### Marketplace (top right) diff --git a/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/sparql-constraints/index.md b/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/sparql-constraints/index.md index f062de3aa..7be926f60 100644 --- a/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/sparql-constraints/index.md +++ b/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/sparql-constraints/index.md @@ -38,7 +38,7 @@ Click the :eccenca-module-marketplace: **Packages** icon in the main menu under Search for the required ontologies / vocabularies, then click the **Install** button to install the "The RDF Schema Vocabulary" and "Friend of a Friend (FOAF)" vocabulary. Wait until each installation is complete (the **Install** button will change to **Uninstall**) before installing the next package. -![Search and install ontologies](bacui-marketplace-schema-search.png){ class="bordered" } +![Search and install ontologies](bacui-marketplace-schema-search.png){ class="bordered" width="90%" } ### Create Shape Catalog diff --git a/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/workflow-trigger/index.md b/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/workflow-trigger/index.md index b1bb2432e..6099518f7 100644 --- a/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/workflow-trigger/index.md +++ b/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/workflow-trigger/index.md @@ -36,7 +36,7 @@ A `shui:WidgetIntegration` resource needs to be created from the previously defi Once the widget is created, it can be attached to a Node Shape by using a special-purpose non-validating [Widget property](../node-shapes/index.md#widgets). -![](adding_widget_to_node.png) +![](adding_widget_to_node.png){ width="95%" } ## Payload Structure diff --git a/docs/explore-and-author/workspace-configuration/index.md b/docs/explore-and-author/workspace-configuration/index.md index 639fa07ec..dd31c57a6 100644 --- a/docs/explore-and-author/workspace-configuration/index.md +++ b/docs/explore-and-author/workspace-configuration/index.md @@ -25,7 +25,7 @@ Click the drop-down list and click the Application view you want to open. Click the :eccenca-module-workspace-configuration: **Application view configuration** icon in the main menu. -![Configuration module](configuration.png){ class="bordered" } +![Configuration module](configuration.png){ class="bordered" width="45%" } Select the Application view from the dropdown menu to configure. diff --git a/tasks/backlog.md b/tasks/backlog.md index bd60b1783..f4d3fbced 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -1,6 +1,7 @@ # Backlog: print-on-demand book block -Work breakdown for [spec.md](spec.md). **Status 2026-09-15: P0-P12 done and verified, P13-P17 open.** +Work breakdown for [spec.md](spec.md). **Status 2026-09-15: P0-P14 and P18-P24 done and verified; P15 has its tooling and +42 images to replace or accept; P16 and P17 open.** The previous content of this file (the temporary tag-listing renderer) is in the git history. @@ -246,7 +247,7 @@ edition; the preflight measures the lightest fill. **Done 2026-09-15:** in print, text and alarm accents are black, the orange a dark grey, the peach bands 20 % black; code, code spans and admonitions have no ground - a thin frame marks a code block, the bar on -the left an admonition. The greyscale preview of P13 does not exist yet, so Ghostscript's `pnggray` +the left an admonition. The greyscale preview of P13 did not exist then (it does now: `--gray`), so Ghostscript's `pnggray` rendered all 962 pages instead: on the 610 pages without images no area is lighter than 20 % black, and the bands measure 204 (20 % black) on 36 pages. The part diagrams' red frames are part of their images and stay visible as a dark frame on the covers. @@ -282,7 +283,33 @@ copies in `dist/pdf/print/images/`. The PDF grows from 72 to 165 MB with the los --- -## P13 - Normalization pass (optional) +## P13 - Normalization pass (optional) - **done** + +Done 2026-09-15: `dec-tool pdf-normalize` (`tools/pdf_normalize.py`), run by `task pdf:print -- --normalize` +or PDF_NORMALIZE=1, writes `…-print-x4.pdf`. Refinements: + +- **Profile not vendored:** its copyright reads "All Rights Reserved" (spec §7). It is fetched from the ECI + into `dist/icc/` with a SHA-256 check; `--icc-profile` / PDF_ICC_PROFILE name a copy. +- **No prefix file in the repository:** the pdfmark prefix is generated with the title and the profile's path. +- **Transparency first:** Ghostscript 10.08 segfaults on Typst's colour emoji and leaves out transparent + SVG content (spec §7). The print edition therefore renders the 5 transparent SVGs and the 10 emoji sequences + to PNG with Typst; the style swaps emoji by show rule, in code too. The screen PDF is verified unchanged. +- **Failures are loud:** the run fails on a non-zero exit, on `error executing PDF token` and on a missing + PDF/X marker, and deletes the output. +- **Checked twice:** the build runs the preflight (P14) on the book block before Ghostscript, and on the + PDF/X-4 copy with `--pdfx` after. +- **Greyscale preview:** `task pdf:print -- --gray` (PDF_GRAY=1; `dec-tool pdf-normalize --gray` for an + existing PDF) writes `…-print-gray-x4.pdf`. It is the same pass in DeviceGray, and it serves as a screen + check of the black-and-white print for P11. It combines with `--normalize`. Verified on the 868-page + book block: 181 s, 59 MB, all 556 images grey, A4 unchanged. Since 2026-09-16 it is PDF/X-4 as well, + by the generic `default_gray.icc` of Ghostscript (condition `sGray`, one component, no registry). + Verified on the 872-page book: 176 s, 57 MB, the marker and a one-component intent, all 556 images grey, + and `dec-tool pdf-preflight --pdfx --intent sGray` passes; four unit tests. + +Verified on the full book block: 868 pages, Ghostscript 218 s, 175 MB; the preflight passes with +`--pdfx`: PDF/X-4 marker, FOGRA39 intent, 556 images all CMYK and none above 300 ppi, fonts embedded, no +Type 3 font, no transparency, no annotations. Rendered pages 26 (card icons), 472 (Excalidraw diagram) +and 662 (emoji) match the RGB file. 9 unit tests, 5 more for the rendered emoji and SVGs. `dec-tool pdf-normalize ` (run by `task pdf:print` with `--normalize` / `PDF_NORMALIZE=1`), spec §7: @@ -304,7 +331,18 @@ unchanged; render sample pages before and after and compare. --- -## P14 - Preflight report +## P14 - Preflight report - **done** + +Done 2026-09-15: `dec-tool pdf-preflight [--pdfx] [--max-pages]` (`tools/pdf_preflight.py`). Page +sizes, annotations and fill colours come from pypdf, fonts, images and word boxes from poppler's +`pdffonts`, `pdfimages -list` and `pdftotext -bbox`. Light areas are a **warning** that does not fail the +run - artwork and emoji carry light fills too; every other check is an error. With `--pdfx` it also +requires the PDF/X-4 marker, a FOGRA39 output intent and CMYK or grey images at most 300 ppi. +`build-pdf --edition print` runs it on its final file, the normalized one when P13 ran, and fails on an +error after writing the PDF. Verified: 9 unit tests; the print book block (868 pages) passes every check +in 7 s; the screen PDF fails as it should - 778 pages with annotations, 840 page numbers on the inner +edge, soft masks, light fills. Refinement: a fill of exactly 20 % black (`0.8 g`, 0.19999… in floating +point) counts as passing - the first run reported it on 102 pages. `dec-tool pdf-preflight `: A4 page size, page count at most 1,200 and even, fonts embedded, images below 300 ppi, soft masks, lightest fill below 20 % black, page-number position (P3's check), blank @@ -325,6 +363,13 @@ with a fresh screenshot, or accept it in spec §8. **Verify:** every entry replaced or accepted. **Est:** medium, mostly content. **Depends on:** P12, P24. +Tooling done 2026-09-15: the print build collects every original below 150 ppi at its printed size +(`resolve_images(..., low_resolution)`), leaves out those listed under `accepted-low-resolution` in +`tools/pdf/print.yml`, writes the rest lowest first to `dist/pdf/print/low-resolution.tsv` and prints +their count. Accepting moved from spec §8 to `print.yml`, next to the other print settings. **Open, content +work:** 42 images, 85-149 ppi, most in `consume/populate-data-to-neo4j` (8), `explore-and-author/bke-module` +(5) and `distribution/marketplace` (4). + --- ## P16 - CI @@ -534,7 +579,15 @@ with more than 20 parameters. --- -## P24 - Image widths in the sources +## P24 - Image widths in the sources - **done** + +Done 2026-09-15: `dec-tool image-widths` (`tools/image_widths.py`) checks without `--fix` and fails when +an image lacks a width. Refinement: an image at 100 % or wider gets **no** width - the column caps it in +print anyway, and 456 such images would otherwise have been pinned to the article width on the site, +upscaling the narrower ones there. `--fix` wrote 24 widths into 14 pages (20 % to 95 %), 10 of them on +the IDS/OSINT tutorial, which the print edition omits; rumdl clean. Below 150 ppi in the print edition: +43 before, 42 after - an image without a declared density now prints at 96 instead of 72 ppi, still +below 150, so P15 keeps its list. `task check` does not run the check yet. 15 unit tests. Spec §11, D18. `dec-tool image-widths --check|--fix` writes `{ width="NN%" }` for raster images without a width in pages that are not generated: the pixel width divided by the capture scale and by the full page @@ -550,6 +603,26 @@ sharply than before. --- +## P25 - A heading with a single line at the foot of a page - **done** + +Review finding of 2026-09-16: A.3 Task and Operator Reference sat at the foot of page 29 with its +intended-audience line under it, and the section started on the next page. A heading is sticky, so it is +never last on a page - but a one-line lead satisfies that, and the break falls after the line. + +Done the same day: `keep_lead_with_heading` in `tools/build_pdf.py` (print edition only) wraps a paragraph +that directly follows a heading of level 1 to 4 and is at most 200 characters long in +`div.keep-with-next`; `filter.lua` maps it to `keep-with-next` in `style.typ`, a sticky block, so the lead +carries heading and line to the block that follows. Levels 5 and 6 are left out: they are the operator +entries, which have their own sticky field blocks. + +**Verified:** 304 lead lines wrapped; A.3 now starts a page with its content; across the book, headings +with at most one line under them at a page foot fall from 23 to 6, and the book grows from 868 to 872 +pages. Measured against levels 1 to 3 alone, which leaves 17 of them at 868 pages: the four pages buy +eleven fewer stranded headings, and one new one appears (B.2.2.6). `LEAD_HEADINGS` is the one place to +change if the shorter book matters more. The screen edition does not run the step. One unit test. + +--- + ## Sequencing ```text diff --git a/tasks/spec.md b/tasks/spec.md index 2bf6a81f4..ca4867be1 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -5,7 +5,9 @@ refines this spec. §10, excluding content from the print edition, was decided on 2026-09-15 (D12-D14) and implemented the same day (backlog P18): the print edition has 666 pages. §11 collects the pull request review of 2026-09-15, decided the same day (D15-D18); backlog P19-P24, of which P19-P23 were implemented -on 2026-09-15: the print edition has 870 pages. +on 2026-09-15: the print edition has 870 pages. P13, P14 and P24, and the tooling of P15, followed the same +day: PDF/X-4 normalization (§7, with the transparency Ghostscript cannot convert rendered to images), the +preflight report, image widths in the sources and the list of low-resolution originals; 868 pages. **Branch:** `feature/print-on-demand`, based on `main` at `c20d74b94` (PDF export merged). **Goal:** a *book block* - the interior file of a printed, perfect-bound book - built next to the screen PDF, which BoD accepts without rework. @@ -72,8 +74,8 @@ For comparison, not pursued: Amazon KDP allows 828 pages (black ink on white), L - Page numbers sit on the **outer edge**: left on even (verso) pages, right on odd (recto) pages. - Running titles mirror too, following book convention (the larger unit on the left page): - - verso: `1234` at the outer left, `Part A: Build` beside it (revised after the review, §11) - - recto: `A.3 Task and Operator Reference` beside `1235` at the outer right + - verso: `1234` at the outer left, `Part A: Build` beside it (revised after the review, §11) + - recto: `A.3 Task and Operator Reference` beside `1235` at the outer right - `| total` is dropped - it has no meaning on paper. - No logo and no version stamp on text pages (D9); both stay on the title page and the imprint. - Margins become `inside`/`outside` with `binding: left` on A4 (D7). The gutter must grow with the @@ -246,6 +248,10 @@ About 1,015 pages: within BoD's 1,200 for A4 on 80 g. - The front contents start on page 3, recto. - Blank pages stay empty - no header, no footer, no page number. - The final page count is even. +- A heading never ends a page with a single line under it. Headings are sticky, which keeps one block with + them; a short lead - the intended audience of a section, say - satisfies that and leaves the two of them + at the foot of the page. The build therefore makes a lead of at most 200 characters under a heading of + level 1 to 4 sticky as well (backlog P25), so both move to where the content starts. ### Typography @@ -337,7 +343,27 @@ Traps found: truncated file - supply the profile only through the prefix file, which reads it with `--permit-file-read=`. - Ghostscript ships a generic `default_cmyk.icc`, not FOGRA39. The ISO Coated v2 profile comes from the - ECI and has to be downloaded and vendored (with its licence) or fetched in CI. + ECI. **Not vendored** (checked 2026-09-15): the ECI only says the profiles may be "freely downloaded", + and the profile's own copyright reads "© Heidelberger Druckmaschinen AG. All Rights Reserved". The + normalization fetches `ECI_Offset_2009/ISOcoated_v2_eci.icc` from + `https://www.eci.org/lib/exe/eci_offset_2009.zip` once, checks its SHA-256 + (`128dc02f…94b8`) and caches it in the gitignored `dist/icc/`; `--icc-profile` or `PDF_ICC_PROFILE` + names a local copy instead, for CI without network access. +- **Ghostscript 10.08 cannot convert transparency drawn by Typst** (found on the full book block, + 2026-09-15). Typst writes colour emoji as a Type 3 font whose glyphs carry shadings and soft masks, and + embeds SVGs that use opacity or masks with transparency groups and soft masks: + - with CMYK conversion, the emoji pages make pdfwrite **segfault** (exit -11) when it closes the file - + not on every run, and after all pages, leaving a truncated file + - with `-dPDFX=4`, it reports `error executing PDF token`, calls the error repaired, exits 0 and + **leaves out** what it could not draw: the ⚠️ emoji, most of the CKAN card icon + - `-dNOTRANSPARENCY` avoids the error but ignores the masks: an Excalidraw diagram then prints its + arrow through the label + - `qpdf --check` finds nothing wrong in the input, and without `-dPDFX` and CMYK the pages convert + + So the print edition hands Ghostscript no transparency (backlog P13): the build renders SVGs that use + opacity, masks or filters, and every colour emoji sequence, to PNG on white with Typst, and the style + swaps each emoji sequence for its image with a show rule, in code too. The preflight fails on + transparency and Type 3 fonts, and the normalization fails on a token error instead of trusting exit 0. ### Upsampling to 300 dpi @@ -358,9 +384,12 @@ need replacing (P14). ### Black and white The book prints in black and white, but BoD converts RGB itself and the pass is specified as CMYK. A -greyscale variant (`-sColorConversionStrategy=Gray`, output intent from a grey profile) would show the -final tones on screen and give smaller files. Keep CMYK as specified; add greyscale as a preview mode of -the same task, for checking the palette (P11). +greyscale variant (`-sColorConversionStrategy=Gray`) shows the final tones on screen and gives smaller +files. Kept CMYK as specified; greyscale is a preview mode of the same pass for checking the palette +(P11), built on 2026-09-15 as `--gray`: DeviceGray with the same image handling and failure checks. It is +PDF/X-4 as well (2026-09-16), with the generic grey profile Ghostscript ships, `default_gray.icc`, as its +output intent - one component instead of four, the condition `sGray`, and no registry, since the +condition is not a registered one. `--icc-profile` names another grey profile. ## 8. Acceptance @@ -378,7 +407,9 @@ the same task, for checking the palette (P11). - internal links print page references, external links footnotes; the book block has no link annotations - page count even and at most 1,200 - grey areas at least 20 % black; all fonts embedded; no soft masks left -- every image at 300 ppi or more at its printed size, or listed and accepted +- every image at 300 ppi or more at its printed size; an original below 150 ppi is listed in + `dist/pdf/print/low-resolution.tsv` until it is replaced or accepted under `accepted-low-resolution` in + `print.yml` - with normalization: the file declares PDF/X-4, carries a FOGRA39 output intent, CMYK images only, no image above 300 ppi - a page key in `print.yml` drops that page; `omit` leaves no title and no note for a page or a section @@ -525,7 +556,7 @@ Several blocks - also inside a list item, a content tab or an admonition. ## 11. Review of the first print build **Status:** findings of the pull request review, 2026-09-15, decided the same day as D15-D18 (§4). -P19-P23 are implemented; P24 and P13 are open. +P19-P24 and P13 are implemented; P15 keeps 42 images to replace or accept. | Finding | Today | Proposal | Backlog | | :-- | :-- | :-- | :-- | @@ -534,8 +565,8 @@ P19-P23 are implemented; P24 and P13 are open. | The author order should follow the printed content | commits to the whole repository (§3) | commits to the printed pages (D16) | P21 | | Facing cards in a two-column grid should be equally high | each card as high as its text | one height per row | P22 | | The operator reference should print descriptions and parameters | overview tables, `list` mode (§5) | one compact entry per operator, replacing the overview tables (D17) | P23 | -| PDF/X-4 and CMYK | specified, not built (§7) | unchanged | P13 | -| Low-resolution images mostly lack `width` in the Markdown | an image without `width` prints at its declared density | widths computed from pixels and capture density against the full page width, written into the sources (D18) | P24, then P15 | +| PDF/X-4 and CMYK | specified, not built (§7) | unchanged; done, with transparency rendered to images first | P13 | +| Low-resolution images mostly lack `width` in the Markdown | an image without `width` prints at its declared density | widths computed from pixels and capture density against the full page width, written into the sources (D18); done, 42 originals stay below 150 ppi | P24, then P15 | ### Footer: the part label diff --git a/tests/test_build_pdf_emoji.py b/tests/test_build_pdf_emoji.py new file mode 100644 index 000000000..920b50f30 --- /dev/null +++ b/tests/test_build_pdf_emoji.py @@ -0,0 +1,110 @@ +"""Test what the print edition renders as images for Ghostscript: colour emoji and transparent SVGs (backlog P13)""" +import base64 +import shutil +from collections import Counter +from pathlib import Path + +import pytest +from bs4 import BeautifulSoup +from PIL import Image + +from tools.build_pdf import ( + EMOJI_FONT, + TEXT_FONT, + emoji_images, + emoji_pattern, + font_codepoints, + rasterize_transparent_svgs, + render_emoji, + typst_png, +) + + +def test_the_cmaps_of_the_vendored_fonts_are_read(): + emoji = font_codepoints(EMOJI_FONT) + text = font_codepoints(TEXT_FONT) + assert {0x26A0, 0x1F604, 0x1F1E9} <= emoji + assert {ord("A"), ord("é"), ord("€")} <= text + assert 0x1F604 not in text + + +def test_emoji_sequences_are_found_whole_and_text_is_left_alone(): + pattern = emoji_pattern() + text = "Warn ⚠️, wave 👋🏽, coder 👩‍💻, flag 🇩🇪, smile 😄 - plain text, 42 € and é." + assert pattern.findall(text) == ["⚠️", "👋🏽", "👩‍💻", "🇩🇪", "😄"] + + +def test_emoji_of_text_and_code_are_rendered_longest_first(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + html = '

Warn ⚠️ now 😄

🎤 Type of package
' + doc = BeautifulSoup(html, "html.parser") + rendered = [] + + def fake_render(sequences, targets, typst): + rendered.append(list(sequences)) + assert [target.name for target in targets] == ["26a0-fe0f.png", "1f3a4.png", "1f604.png"] + + stats = Counter() + pairs = emoji_images(doc, Path("emoji"), "typst", stats, render=fake_render) + assert rendered == [["⚠️", "🎤", "😄"]] + assert pairs == [["⚠️", "/emoji/26a0-fe0f.png"], ["🎤", "/emoji/1f3a4.png"], ["😄", "/emoji/1f604.png"]] + assert str(doc) == html # the style swaps them; the document keeps its text + assert stats["emoji rendered as images"] == 3 + assert emoji_images(BeautifulSoup("

No emoji here, 42 €.

", "html.parser"), Path("emoji"), "typst", stats) == [] + + +def svg_uri(markup): + return "data:image/svg+xml;base64," + base64.b64encode(markup.encode()).decode() + + +def test_svgs_that_draw_with_transparency_become_png(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + (tmp_path / "site").mkdir() + (tmp_path / "site/masked.svg").write_text('') + (tmp_path / "site/plain.svg").write_text('') + translucent = svg_uri('') + solid = svg_uri('') + doc = BeautifulSoup( + f'' + f'', + "html.parser", + ) + calls = [] + + def fake_render(pages, targets, typst, preamble, what): + calls.append((pages, [target.parent.name for target in targets])) + + stats = Counter() + rasterize_transparent_svgs(doc, Path("svg"), "typst", stats, render=fake_render) + sources = [img["src"] for img in doc.find_all("img")] + assert sources[0].startswith("/svg/masked-") and sources[0].endswith(".png") + assert sources[1] == "/site/plain.svg" + assert sources[2].startswith("/svg/") and sources[2].endswith(".png") + assert sources[3] == solid + assert sources[4] == "/site/shot.png" + pages, directories = calls[0] + assert len(pages) == 2 and '#image("/site/masked.svg")' in pages and directories == ["svg", "svg"] + assert stats["SVG images with transparency rendered as PNG"] == 2 + + +@pytest.mark.skipif(shutil.which("typst") is None, reason="Typst is not installed") +def test_typst_renders_an_emoji_and_an_svg_on_white(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + emoji = Path("out/26a0-fe0f.png") + render_emoji(["⚠️"], [emoji], "typst") + with Image.open(emoji) as image: + assert image.mode == "RGB" + # 10 pt from ascender to descender, 1.172 em, at 600 ppi. + assert abs(image.height - 98) <= 2 + assert image.convert("L").getextrema()[0] < 128 + Path("icon.svg").write_text( + '' + '' + ) + svg = Path("out/icon.png") + typst_png(['#image("/icon.svg")'], [svg], "typst", "", "typst (SVG)") + with Image.open(svg) as image: + assert image.mode == "RGB" + assert abs(image.width - 450) <= 2 # 72 CSS px are 54 pt, at 600 ppi + assert 100 < image.convert("L").getpixel((225, 112)) < 160 # half black on white + assert not list(Path("out").glob("page-*.png")) diff --git a/tests/test_build_pdf_print.py b/tests/test_build_pdf_print.py index d4c4cd1ae..d88d7041f 100644 --- a/tests/test_build_pdf_print.py +++ b/tests/test_build_pdf_print.py @@ -1,12 +1,16 @@ """Test the print edition's section modes - full, list and omit - and its excluded parts of a page""" +from collections import Counter + import click import pytest +from bs4 import BeautifulSoup from tools.build_pdf import ( Generated, NavEntry, SectionRule, apply_section_rules, + keep_lead_with_heading, load_section_rules, md_to_built_html, merge_pages, @@ -355,3 +359,21 @@ def test_a_list_table_summarizes_a_page_by_its_first_paragraph_that_prints(tmp_p doc, _, _ = merge_pages([NavEntry(2, generated=table)], site, "https://example.org/26.2/", print_edition=True) rows = [[cell.get_text(" ", strip=True) for cell in row.find_all(["th", "td"])] for row in doc.find_all("tr")] assert rows == [["Release", "Summary"], ["Corporate Memory 26.2.1", "The second release."]] + + +def test_a_short_lead_line_is_kept_with_its_heading(): + doc = BeautifulSoup( + '

Reference

Intended audience: Experts

cards
' + f'

Details

{"word " * 60}

' + '

A step

Short, under a heading the rule still covers.

' + '
A small heading

Short, but not under one of the larger headings.

' + "

A paragraph that follows no heading.

", + "html.parser", + ) + stats = Counter() + keep_lead_with_heading(doc, stats) + assert [div.p.get_text(" ", strip=True) for div in doc.select("div.keep-with-next")] == [ + "Intended audience: Experts", + "Short, under a heading the rule still covers.", + ] + assert stats["lead lines kept with their heading"] == 2 diff --git a/tests/test_build_pdf_resolution.py b/tests/test_build_pdf_resolution.py new file mode 100644 index 000000000..180e191fb --- /dev/null +++ b/tests/test_build_pdf_resolution.py @@ -0,0 +1,60 @@ +"""Test the list of low-resolution originals the print edition reports (backlog P15)""" +from collections import Counter +from pathlib import Path + +import click +import pytest +from bs4 import BeautifulSoup +from PIL import Image + +from tools.build_pdf import load_accepted_low_resolution, resolve_images, unaccepted_low_resolution + + +def test_originals_below_150_ppi_at_their_printed_size_are_listed_once(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + site = Path("site") + (site / "build").mkdir(parents=True) + Image.new("RGB", (300, 100), "white").save(site / "build/small.png") + Image.new("RGB", (2000, 100), "white").save(site / "build/wide.png") + Image.new("RGB", (600, 100), "white").save(site / "build/half.png") + doc = BeautifulSoup( + '

' + '

', + "html.parser", + ) + low: dict[str, int] = {} + resolve_images(doc, site, Counter(), [], Path("print-images"), low_resolution=low) + # 300 px without a declared density print at 300 pt, 72 ppi; 2000 px fill the + # 16 cm column at 317 ppi; 600 px at 50 % of the column print at 190 ppi. + assert low == {"build/small.png": 72} + + +def test_the_screen_edition_lists_nothing(tmp_path, monkeypatch): + monkeypatch.chdir(tmp_path) + site = Path("site") + (site / "build").mkdir(parents=True) + Image.new("RGB", (300, 100), "white").save(site / "build/small.png") + doc = BeautifulSoup('', "html.parser") + low: dict[str, int] = {} + resolve_images(doc, site, Counter(), [], None, low_resolution=low) + assert low == {} + + +def test_accepted_originals_are_not_listed(tmp_path): + path = tmp_path / "print.yml" + path.write_text("accepted-low-resolution:\n - build/small.png\n") + accepted = load_accepted_low_resolution(path) + assert accepted == {"build/small.png"} + assert unaccepted_low_resolution({"build/small.png": 72, "build/tiny.png": 40, "build/mid.png": 120}, accepted) == [ + ("build/tiny.png", 40), + ("build/mid.png", 120), + ] + + +def test_without_an_accepted_list_nothing_is_accepted(tmp_path): + path = tmp_path / "print.yml" + path.write_text("body-weight: light\n") + assert load_accepted_low_resolution(path) == set() + path.write_text("accepted-low-resolution: build/small.png\n") + with pytest.raises(click.ClickException, match="accepted-low-resolution"): + load_accepted_low_resolution(path) diff --git a/tests/test_image_widths.py b/tests/test_image_widths.py new file mode 100644 index 000000000..bcf60449c --- /dev/null +++ b/tests/test_image_widths.py @@ -0,0 +1,80 @@ +"""Test the screenshot widths written into the Markdown sources (tasks/spec.md, §11, D18)""" +import pytest +from PIL import Image + +from tools.image_widths import add_widths, capture_scale, width_percent + + +@pytest.mark.parametrize( + "dpi, scale", + [(None, 1.0), (72, 1.0), (96, 1.0), (120, 1.25), (144, 2.0), (192, 2.0)], +) +def test_the_capture_scale_follows_the_declared_density(dpi, scale): + assert capture_scale(dpi) == scale + + +@pytest.mark.parametrize( + "pixels, dpi, percent", + [ + (605, None, 100), # the full page width at scale 1 + (300, None, 50), # 49.6 % rounds to 50 + (1210, 144, 100), # a Retina capture: 605 CSS pixels + (400, 144, 35), # 200 CSS pixels: 33 % rounds to 35 + (480, 120, 65), # 384 CSS pixels: 63.5 % rounds to 65 + (20, None, 5), # never below 5 % + (2000, None, 100), # never above 100 % + ], +) +def test_the_width_is_the_natural_width_as_a_share_of_the_full_page(pixels, dpi, percent): + assert width_percent(pixels, dpi) == percent + + +def picture(path, width, dpi=None): + image = Image.new("RGB", (width, 10), "white") + if dpi: + image.save(path, dpi=(dpi, dpi)) + else: + image.save(path) + + +def test_images_without_a_width_get_one_and_the_rest_stay_as_they_are(tmp_path): + picture(tmp_path / "plain.png", 300) + picture(tmp_path / "framed.png", 600, dpi=144) + picture(tmp_path / "sized.png", 300) + picture(tmp_path / "titled.png", 400) + picture(tmp_path / "wide.png", 1500) + picture(tmp_path / "coded.png", 300) + (tmp_path / "diagram.svg").write_text("") + text = ( + "# Page\n\n" + "![Plain](plain.png)\n\n" + '![Framed](framed.png){ class="bordered" }\n\n' + '![Sized](sized.png){ class="bordered" width="70%" }\n\n' + '![Titled](titled.png "A title")\n\n' + "![Wide](wide.png)\n\n" + "![Remote](https://example.org/remote.png)\n\n" + "![Vector](diagram.svg)\n\n" + "![Missing](gone.png)\n\n" + "```markdown\n![Coded](coded.png)\n```\n" + ) + result, changes = add_widths(text, tmp_path) + # A full-width image gets no width: it fills the column in print either way. + assert result == ( + "# Page\n\n" + '![Plain](plain.png){ width="50%" }\n\n' + '![Framed](framed.png){ class="bordered" width="50%" }\n\n' + '![Sized](sized.png){ class="bordered" width="70%" }\n\n' + '![Titled](titled.png "A title"){ width="65%" }\n\n' + "![Wide](wide.png)\n\n" + "![Remote](https://example.org/remote.png)\n\n" + "![Vector](diagram.svg)\n\n" + "![Missing](gone.png)\n\n" + "```markdown\n![Coded](coded.png)\n```\n" + ) + assert changes == [("plain.png", 50), ("framed.png", 50), ("titled.png", 65)] + + +def test_a_page_whose_images_all_have_widths_is_unchanged(tmp_path): + picture(tmp_path / "sized.png", 300) + text = '![Sized](sized.png){ width="40%" }\n' + assert add_widths(text, tmp_path) == (text, []) diff --git a/tests/test_pdf_normalize.py b/tests/test_pdf_normalize.py new file mode 100644 index 000000000..c3aeff897 --- /dev/null +++ b/tests/test_pdf_normalize.py @@ -0,0 +1,192 @@ +"""Test the Ghostscript pass that turns the print edition into PDF/X-4 (tasks/spec.md, §7; backlog P13)""" +import glob +import hashlib +import shutil +import zipfile +from pathlib import Path + +import click +import pytest +from pypdf import PdfReader, PdfWriter +from pypdf.generic import ArrayObject, DecodedStreamObject, DictionaryObject, NameObject, TextStringObject + +from tools.pdf_normalize import ( + ensure_gray_profile, + fetch_profile, + ghostscript_command, + gray_command, + gray_path, + gray_preview, + is_pdfx4, + normalize, + normalized_path, + pdfx_prefix, + postscript_string, +) + + +def test_postscript_strings_escape_backslashes_and_parentheses(): + assert postscript_string(r"a (b) \c") == r"(a \(b\) \\c)" + + +def test_the_prefix_declares_pdfx4_and_the_fogra39_output_intent(): + prefix = pdfx_prefix(Path("/profiles/ISO (coated).icc"), "Corporate Memory (26.2)") + assert "/GTS_PDFXVersion (PDF/X-4)" in prefix + assert "/Trapped /False" in prefix + assert r"/Title (Corporate Memory \(26.2\))" in prefix + assert r"(/profiles/ISO \(coated\).icc) (r) file" in prefix + assert "/OutputConditionIdentifier (FOGRA39)" in prefix + assert "/N 4" in prefix + + +def test_the_ghostscript_command_follows_the_verified_recipe(): + command = ghostscript_command("gs", Path("prefix.ps"), Path("profile.icc"), Path("in.pdf"), Path("out.pdf")) + for flag in ( + "-dPDFX=4", "-sDEVICE=pdfwrite", "-sColorConversionStrategy=CMYK", "-sProcessColorModel=DeviceCMYK", + "-dColorImageResolution=300", "-dGrayImageResolution=300", "-dColorImageDownsampleThreshold=1.0", + "-dPreserveAnnots=false", "--permit-file-read=profile.icc", "-sOutputFile=out.pdf", + ): + assert flag in command + # The profile goes in through the prefix file only: -sOutputICCProfile with -dPDFX crashes Ghostscript. + assert not any(part.startswith("-sOutputICCProfile") for part in command) + assert command[-2:] == ["prefix.ps", "in.pdf"] + + +def test_the_gray_preview_is_the_same_pass_in_devicegray(): + command = gray_command("gs", Path("prefix.ps"), Path("gray.icc"), Path("in.pdf"), Path("out.pdf")) + for flag in ( + "-dPDFX=4", "-sDEVICE=pdfwrite", "-sColorConversionStrategy=Gray", "-sProcessColorModel=DeviceGray", + "-dColorImageResolution=300", "-dPreserveAnnots=false", "--permit-file-read=gray.icc", + "-sOutputFile=out.pdf", + ): + assert flag in command + assert not any(part.startswith("-sOutputICCProfile") for part in command) + assert command[-2:] == ["prefix.ps", "in.pdf"] + + +def test_each_output_is_named_after_what_it_is(): + book = Path("dist/documentation-eccenca-com-26-2-print.pdf") + assert normalized_path(book).name == "documentation-eccenca-com-26-2-print-x4.pdf" + assert gray_path(book).name == "documentation-eccenca-com-26-2-print-gray-x4.pdf" + + +def test_the_grey_prefix_names_one_component_and_no_registry(): + prefix = pdfx_prefix( + Path("/profiles/gray.icc"), "Book", components=1, identifier="sGray", condition="A grey profile", + registry=None, + ) + assert "/N 1" in prefix + assert "/OutputConditionIdentifier (sGray)" in prefix + assert "/Info (A grey profile)" in prefix + assert "/RegistryName" not in prefix + + +def blank_pdf(path, pdfx=False): + writer = PdfWriter() + writer.add_blank_page(595.28, 841.89) + if pdfx: + writer.add_metadata({"/GTS_PDFXVersion": "PDF/X-4"}) + intent = DictionaryObject({ + NameObject("/Type"): NameObject("/OutputIntent"), + NameObject("/S"): NameObject("/GTS_PDFX"), + NameObject("/OutputConditionIdentifier"): TextStringObject("FOGRA39"), + }) + writer.root_object[NameObject("/OutputIntents")] = ArrayObject([intent]) + writer.write(path) + return path + + +def test_only_a_file_with_the_marker_and_an_output_intent_counts_as_pdfx4(tmp_path): + assert is_pdfx4(blank_pdf(tmp_path / "x4.pdf", pdfx=True)) + assert not is_pdfx4(blank_pdf(tmp_path / "plain.pdf")) + + +def test_the_profile_is_fetched_from_its_archive_and_checked(tmp_path): + payload = b"icc profile bytes" + archive = tmp_path / "profiles.zip" + member = "ECI_Offset_2009/ISOcoated_v2_eci.icc" + with zipfile.ZipFile(archive, "w") as bundle: + bundle.writestr(member, payload) + target = tmp_path / "cache" / "ISOcoated_v2_eci.icc" + fetch_profile(target, archive.as_uri(), member, hashlib.sha256(payload).hexdigest()) + assert target.read_bytes() == payload + with pytest.raises(click.ClickException, match="checksum"): + fetch_profile(tmp_path / "other.icc", archive.as_uri(), member, "0" * 64) + assert not (tmp_path / "other.icc").exists() + + +def ghostscript_cmyk_profile(): + """The generic CMYK profile Ghostscript ships, where a common install puts it.""" + patterns = ( + "/opt/homebrew/share/ghostscript/iccprofiles/default_cmyk.icc", + "/opt/homebrew/Cellar/ghostscript/*/share/ghostscript/iccprofiles/default_cmyk.icc", + "/usr/local/share/ghostscript/iccprofiles/default_cmyk.icc", + "/usr/share/ghostscript/*/iccprofiles/default_cmyk.icc", + "/usr/share/color/icc/ghostscript/default_cmyk.icc", + ) + found = [path for pattern in patterns for path in glob.glob(pattern)] + return Path(found[0]) if found else None + + +def test_a_failed_ghostscript_run_leaves_no_file_and_names_the_page(tmp_path): + fake = tmp_path / "gs" + fake.write_text( + "#!/bin/sh\n" + 'for arg in "$@"; do case "$arg" in -sOutputFile=*) printf partial > "${arg#-sOutputFile=}";; esac; done\n' + "echo 'Page 1'; echo 'Page 2'\n" + "echo ' **** Error: error executing PDF token' >&2\n" + "exit 1\n" + ) + fake.chmod(0o755) + output = tmp_path / "out.pdf" + with pytest.raises(click.ClickException, match="after page 2") as failure: + normalize(blank_pdf(tmp_path / "in.pdf"), output, tmp_path / "profile.icc", str(fake), title="Test") + assert "error executing PDF token" in failure.value.message + assert not output.exists() + + +def test_content_ghostscript_repaired_away_fails_the_run(tmp_path): + fake = tmp_path / "gs" + fake.write_text( + "#!/bin/sh\n" + 'for arg in "$@"; do case "$arg" in -sOutputFile=*) printf repaired > "${arg#-sOutputFile=}";; esac; done\n' + "printf 'The following errors were encountered at least once while processing this file:\\n" + "\\terror executing PDF token\\n' >&2\n" + ) + fake.chmod(0o755) + output = tmp_path / "out.pdf" + with pytest.raises(click.ClickException, match="left out content"): + normalize(blank_pdf(tmp_path / "in.pdf"), output, tmp_path / "profile.icc", str(fake), title="Test") + assert not output.exists() + + +@pytest.mark.skipif(shutil.which("gs") is None, reason="Ghostscript is not installed") +def test_ghostscript_writes_pdfx4(tmp_path): + profile = ghostscript_cmyk_profile() + if profile is None: + pytest.skip("no CMYK profile of Ghostscript found") + output = tmp_path / "out.pdf" + normalize(blank_pdf(tmp_path / "in.pdf"), output, profile, "gs", title="Test (1)") + assert is_pdfx4(output) + + +@pytest.mark.skipif(shutil.which("gs") is None, reason="Ghostscript is not installed") +def test_ghostscript_writes_a_gray_preview_as_pdfx4(tmp_path): + try: + ensure_gray_profile() + except click.ClickException: + pytest.skip("no grey ICC profile of Ghostscript found") + writer = PdfWriter() + page = writer.add_blank_page(595.28, 841.89) + content = DecodedStreamObject() + content.set_data(b"1 0 0 rg 0 0 100 100 re f 0 1 1 0 k 100 100 100 100 re f") + page[NameObject("/Contents")] = writer._add_object(content) + source = tmp_path / "colour.pdf" + writer.write(source) + output = gray_preview(source, tmp_path / "colour-gray.pdf", title="Test") + data = PdfReader(output).pages[0].get_contents().get_data() + assert b"re" in data # the areas are still drawn + assert b" rg" not in data and b" k\n" not in data and b" k " not in data + assert is_pdfx4(output) + intents = PdfReader(output).trailer["/Root"]["/OutputIntents"] + assert str(intents[0].get_object()["/OutputConditionIdentifier"]) == "sGray" diff --git a/tests/test_pdf_preflight.py b/tests/test_pdf_preflight.py new file mode 100644 index 000000000..a8d1bd6be --- /dev/null +++ b/tests/test_pdf_preflight.py @@ -0,0 +1,181 @@ +"""Test the preflight report of the print edition (tasks/spec.md, §8; backlog P14)""" +from pypdf import PdfWriter +from pypdf.annotations import Link +from pypdf.generic import ( + ArrayObject, + DecodedStreamObject, + DictionaryObject, + FloatObject, + NameObject, + NumberObject, + RectangleObject, + TextStringObject, +) + +from tools.pdf_preflight import ( + area_tints, + check_annotations, + check_blank_pages, + check_fonts, + check_images, + check_light_areas, + check_page_count, + check_page_numbers, + check_page_sizes, + check_pdfx, + check_soft_masks, + check_transparency, + parse_pdffonts, + parse_pdfimages, + read_pdf, + read_transparency, +) + +A4 = (595.28, 841.89) + +PDFFONTS = """\ +name type encoding emb sub uni object ID +------------------------------------ ----------------- ---------------- --- --- --- --------- +ABCDEF+Roboto-Light CID TrueType Identity-H yes yes yes 9 0 +Helvetica Type 1 Standard no no no 12 0 +""" + +PDFIMAGES = """\ +page num type width height color comp bpc enc interp object ID x-ppi y-ppi size ratio +-------------------------------------------------------------------------------------------- + 1 0 image 1890 317 rgb 3 8 image no 9 0 300 300 42.1K 2.3% + 2 1 image 500 300 rgb 3 8 image no 12 0 150 150 10.0K 2.2% + 2 2 smask 500 300 gray 1 8 image no 13 0 150 150 1.0K 0.1% + 3 3 image 1890 317 cmyk 4 8 image no 15 0 301 301 42.1K 2.3% + 4 4 image 49 49 cmyk 4 8 image no 16 0 294 294 1.0K 2.3% + 4 5 image 40 40 cmyk 4 8 image no 17 0 250 250 1.0K 2.3% +""" + + +def test_every_page_count_and_size_problem_is_named(): + assert check_page_count(668, 1200).passed + assert check_page_count(669, 1200).problems == ["669 pages - a printed book needs an even count"] + assert check_page_count(1202, 1200).problems == ["1202 pages - more than 1200"] + assert check_page_sizes([A4, (612.0, 792.0), A4]).problems == ["page 2: 612 x 792 pt, not A4"] + + +def test_fonts_that_are_not_embedded_fail(): + rows = parse_pdffonts(PDFFONTS) + assert [row["name"] for row in rows] == ["ABCDEF+Roboto-Light", "Helvetica"] + assert check_fonts(rows).problems == ["Helvetica (Type 1) is not embedded"] + + +def test_type3_fonts_fail(): + rows = parse_pdffonts(PDFFONTS + "NotoColorEmoji Type 3 Custom yes no yes 24 0\n") + assert check_fonts(rows).problems[-1].startswith("NotoColorEmoji is a Type 3 font") + + +def test_images_below_300_ppi_and_soft_masks_fail(): + rows = parse_pdfimages(PDFIMAGES) + assert len(rows) == 6 + # 49 pixels at 294 ppi are one pixel short of 300 - rounding; 40 at 250 are not. + assert check_images(rows, pdfx=False).problems == ["page 2: image 1 at 150 ppi", "page 4: image 5 at 250 ppi"] + assert check_soft_masks(rows).problems == ["page 2: soft mask on image 2"] + + +def test_with_pdfx_images_are_cmyk_or_grey_and_at_most_300_ppi(): + problems = check_images(parse_pdfimages(PDFIMAGES), pdfx=True).problems + assert "page 1: image 0 in rgb, not CMYK" in problems + assert "page 3: image 3 at 301 ppi" not in problems # a pixel of rounding is fine + + +def test_the_output_intent_must_name_the_condition_the_edition_prints_by(tmp_path): + writer = PdfWriter() + writer.add_blank_page(*A4) + writer.add_metadata({"/GTS_PDFXVersion": "PDF/X-4"}) + intent = DictionaryObject({ + NameObject("/Type"): NameObject("/OutputIntent"), + NameObject("/S"): NameObject("/GTS_PDFX"), + NameObject("/OutputConditionIdentifier"): TextStringObject("sGray"), + }) + writer.root_object[NameObject("/OutputIntents")] = ArrayObject([intent]) + path = tmp_path / "gray.pdf" + writer.write(path) + assert check_pdfx(path, "sGray").passed + assert check_pdfx(path).problems == ["output intent sGray, not FOGRA39"] + + +def test_transparency_in_page_resources_fails(tmp_path): + writer = PdfWriter() + page = writer.add_blank_page(*A4) + form = DecodedStreamObject() + form.set_data(b"0 0 1 1 re f") + form.update({ + NameObject("/Type"): NameObject("/XObject"), + NameObject("/Subtype"): NameObject("/Form"), + NameObject("/BBox"): ArrayObject([NumberObject(0), NumberObject(0), NumberObject(1), NumberObject(1)]), + NameObject("/Group"): DictionaryObject({NameObject("/S"): NameObject("/Transparency")}), + }) + page[NameObject("/Resources")] = DictionaryObject({ + NameObject("/ExtGState"): DictionaryObject({ + NameObject("/G0"): DictionaryObject({NameObject("/ca"): FloatObject(0.5)}), + NameObject("/G1"): DictionaryObject({NameObject("/ca"): FloatObject(1)}), + }), + NameObject("/XObject"): DictionaryObject({NameObject("/X0"): writer._add_object(form)}), + }) + writer.add_blank_page(*A4) + path = tmp_path / "transparent.pdf" + writer.write(path) + found = read_transparency(path) + assert found == {1: {"alpha", "transparency group"}, 2: set()} + assert check_transparency(found).problems == ["page 1: alpha, transparency group"] + + +def test_annotations_fail(): + assert check_annotations([0, 2, 0]).problems == ["page 2: 2 annotations"] + + +def test_page_numbers_sit_on_the_outer_edge(): + words = { + 2: [("2", 60.0, 800.0, 66.0, 812.0)], + 3: [("3", 530.0, 800.0, 536.0, 812.0)], + 4: [("4", 530.0, 800.0, 536.0, 812.0)], + 5: [("Cover", 100.0, 100.0, 150.0, 120.0)], + } + check = check_page_numbers(words, {n: A4 for n in words}) + assert check.problems == ["page 4: page number on the inner edge"] + + +def test_a_blank_page_carries_no_page_furniture(): + words = { + 5: [("5", 530.0, 800.0, 536.0, 812.0), ("Part", 60.0, 800.0, 80.0, 812.0)], + 6: [("Text", 60.0, 100.0, 90.0, 112.0), ("6", 60.0, 800.0, 66.0, 812.0)], + 7: [("7", 530.0, 800.0, 536.0, 812.0)], + } + painted = {5: False, 6: False, 7: True} + check = check_blank_pages(words, {n: A4 for n in words}, painted) + assert check.problems == ["page 5: only page furniture"] + + +def test_light_area_fills_are_found_and_text_is_ignored(): + content = ( + b"q 0.95 g 0 0 10 10 re f Q 0.5 g 0 0 5 5 re f 1 1 1 rg 0 0 1 1 re f " + b"/Cs1 cs 0.9 0.9 0.9 scn 0 0 2 2 re f BT /F1 9 Tf (0.95 g 0 0 1 1 re f) Tj ET " + b"0 0 0 0.1 k 0 0 3 3 re f*" + ) + tints, painted = area_tints(content) + assert tints == {0.05, 0.1} + assert painted + assert area_tints(b"BT /F1 10 Tf (hello) Tj ET") == (set(), False) + assert area_tints(b"0.8 g 0 0 1 1 re f 0.8 0.8 0.8 rg 0 0 1 1 re f") == (set(), True) # exactly 20 % + check = check_light_areas({3: {0.05}, 4: set(), 7: {0.05, 0.1}}) + assert check.level == "warning" + assert check.problems == ["5 % black on pages 3, 7", "10 % black on page 7"] + + +def test_the_pdf_is_read_page_by_page(tmp_path): + writer = PdfWriter() + writer.add_blank_page(*A4) + writer.add_blank_page(612, 792) + writer.add_annotation(1, Link(rect=RectangleObject((10, 10, 20, 20)), target_page_index=0)) + path = tmp_path / "book.pdf" + writer.write(path) + sizes, annotations, contents = read_pdf(path) + assert [tuple(round(value) for value in size) for size in sizes] == [(595, 842), (612, 792)] + assert annotations == [0, 1] + assert len(contents) == 2 diff --git a/tools/README.md b/tools/README.md index 8172defd8..e77ed611d 100644 --- a/tools/README.md +++ b/tools/README.md @@ -12,7 +12,10 @@ supported entry points and pass the right options. | `build-navigation` | `task update:navigation`, `task check:navigation` | Build `nav.yml` from the `docs/**/.pages` files; `--check` diffs instead of writing and fails on drift | | `build-pdf` | `task pdf`, `task pdf:print` | Merge the built `site/` along `nav.yml` into one document and typeset it as a single PDF with pandoc and Typst; style, filter and fonts are in `pdf/` | | `pdf-authors` | `task pdf:authors` | Write the print edition's author list `pdf/authors.yml` from the commits to the printed pages and the names on the authors' GitHub profiles; names and exclusions are kept in `pdf/print.yml`, where it adds each new author without a name | +| `pdf-normalize` | `task pdf:print -- --normalize`, `task pdf:print -- --gray` | Write the print PDF as PDF/X-4 in CMYK with Ghostscript, checking the result for the PDF/X marker; the ISO Coated v2 profile is fetched into `dist/icc/`. `--gray` writes a greyscale preview of the black-and-white print instead, PDF/X-4 by the grey profile Ghostscript ships | +| `pdf-preflight` | `task pdf:print` | Check the book block for print: A4, even page count, embedded fonts, image resolution, soft masks, annotations, page-number position, blank pages, light grey areas | | `check-zensical-output` | `task check:output` | Inspect the built `site/` and fail if a feature we reimplemented for Zensical regressed | +| `image-widths` | - | List raster images without a width in pages that are not generated; `--fix` writes `width="NN%"` from pixels and capture density | | `localize-bundle-assets` | `task build` | Rewrite the third-party asset URLs Zensical bakes into its JavaScript bundle to the vendored copies | | `update-icons` | `task update:icons` | Fetch the eccenca icon set from the gui-elements repository | | `update-di-reference` | `task update:di-reference` | Generate the task and operator reference pages from a running Corporate Memory | diff --git a/tools/__init__.py b/tools/__init__.py index b38c1f2a2..495b3c680 100644 --- a/tools/__init__.py +++ b/tools/__init__.py @@ -5,8 +5,11 @@ from tools.build_navigation import build_navigation from tools.build_pdf import build_pdf from tools.check_zensical_output import check_zensical_output +from tools.image_widths import image_widths from tools.localize_bundle_assets import localize_bundle_assets from tools.pdf_authors import pdf_authors +from tools.pdf_normalize import pdf_normalize +from tools.pdf_preflight import pdf_preflight from tools.publish import publish from tools.update_di_reference import update_di_reference from tools.update_icons import update_icons @@ -19,8 +22,11 @@ def cli(): cli.add_command(build_navigation) cli.add_command(build_pdf) cli.add_command(check_zensical_output) +cli.add_command(image_widths) cli.add_command(localize_bundle_assets) cli.add_command(pdf_authors) +cli.add_command(pdf_normalize) +cli.add_command(pdf_preflight) cli.add_command(publish) cli.add_command(update_icons) cli.add_command(update_di_reference) diff --git a/tools/build_pdf.py b/tools/build_pdf.py index a32534167..e9269b273 100644 --- a/tools/build_pdf.py +++ b/tools/build_pdf.py @@ -25,6 +25,7 @@ import json import re import shutil +import struct import subprocess import sys import time @@ -32,6 +33,7 @@ from collections import Counter from dataclasses import dataclass, field from datetime import date +from functools import lru_cache from pathlib import Path import click @@ -41,6 +43,8 @@ from PIL import Image from tools.pdf_authors import load_imprint_names +from tools.pdf_normalize import ensure_profile, gray_path, gray_preview, normalize as normalize_pdf, normalized_path +from tools.pdf_preflight import print_report, run_preflight DEFAULT_OUT_STEM = "dist/documentation-eccenca-com" SITE_DIR = Path("site") @@ -114,6 +118,21 @@ # 300 dpi at their printed size (tasks/spec.md, §2). TEXT_WIDTH_PT = 16 / 2.54 * 72 PRINT_PPI = 300 +# Originals below this density at their printed size need replacing or accepting (backlog P15). +LOW_RESOLUTION_PPI = 150 +# Colour emoji in the print edition: rendered as images from the vendored font, +# for text the body font does not cover. +EMOJI_FONT = PDF_ASSETS / "fonts" / "noto-color-emoji" / "Noto-COLRv1.ttf" +TEXT_FONT = PDF_ASSETS / "fonts" / "roboto" / "Roboto-Light.ttf" +# What the print edition renders to PNG with Typst: colour emoji and SVGs that +# draw with transparency - opacity, masks, filters, translucent colours. +RENDER_PPI = 600 +MAX_RENDER_WIDTH = round(16 / 2.54 * RENDER_PPI) +SVG_TRANSPARENCY = re.compile(r"opacity| Path: return target +def printed_density(source: Path, width: str | None) -> int: + """An original image's pixel density at the width the print edition prints it, in ppi.""" + with Image.open(source) as image: + return round(image.width / (printed_width_pt(image, width) / 72)) + + +def load_accepted_low_resolution(path: Path = PRINT_YML) -> set[str]: + """The low-resolution originals print.yml accepts as they are, by their path under docs/ (backlog P15).""" + data = yaml.safe_load(path.read_text(encoding="utf-8")) or {} + accepted = data.get("accepted-low-resolution") or [] + if not isinstance(accepted, list) or not all(isinstance(image, str) for image in accepted): + raise click.ClickException(f"{path}: accepted-low-resolution is a list of image paths under docs/") + return set(accepted) + + +def unaccepted_low_resolution(low_resolution: dict[str, int], accepted: set[str]) -> list[tuple[str, int]]: + """The low-resolution originals that are not accepted, lowest density first.""" + return sorted( + ((image, ppi) for image, ppi in low_resolution.items() if image not in accepted), + key=lambda item: (item[1], item[0]), + ) + + def resolve_images( - doc: BeautifulSoup, site_dir: Path, stats: Counter, warnings: list[str], print_images: Path | None = None + doc: BeautifulSoup, + site_dir: Path, + stats: Counter, + warnings: list[str], + print_images: Path | None = None, + low_resolution: dict[str, int] | None = None, ) -> None: """Point images at the files Typst embeds, and replace what it cannot embed. @@ -1286,7 +1333,9 @@ def resolve_images( become their alt text: the build does not fetch from the network. An SVG that keeps its text in foreignObject elements, as Mermaid does by default, renders without that text in Typst, so it is reported. With `print_images`, - raster images point at their print copies in that directory (`print_image`). + raster images point at their print copies in that directory (`print_image`), + and `low_resolution` collects the originals below 150 ppi at their printed + size, by their path under docs/, at their lowest density. """ for lightbox in doc.select("a.glightbox"): lightbox.unwrap() @@ -1314,6 +1363,11 @@ def resolve_images( if path.suffix.lower() == ".svg" and " None: section.unwrap() -def normalize(doc: BeautifulSoup, site_dir: Path, print_images: Path | None = None) -> tuple[Counter, list[str]]: +def font_codepoints(path: Path) -> set[int]: + """The code points a TrueType font maps to a glyph, read from its Windows cmap subtable (format 12 or 4).""" + data = path.read_bytes() + tables = {} + for index in range(struct.unpack(">H", data[4:6])[0]): + entry = 12 + 16 * index + tables[data[entry:entry + 4]] = struct.unpack(">I", data[entry + 8:entry + 12])[0] + cmap = tables[b"cmap"] + subtables = {} + for index in range(struct.unpack(">H", data[cmap + 2:cmap + 4])[0]): + platform, encoding, offset = struct.unpack(">HHI", data[cmap + 4 + 8 * index:cmap + 12 + 8 * index]) + subtables[(platform, encoding)] = cmap + offset + codepoints: set[int] = set() + table = subtables.get((3, 10)) + if table is not None and struct.unpack(">H", data[table:table + 2])[0] == 12: + for group in range(struct.unpack(">I", data[table + 12:table + 16])[0]): + start, end, _ = struct.unpack(">III", data[table + 16 + 12 * group:table + 28 + 12 * group]) + codepoints.update(range(start, end + 1)) + return codepoints + table = subtables[(3, 1)] + segments = struct.unpack(">H", data[table + 6:table + 8])[0] // 2 + ends = struct.unpack(f">{segments}H", data[table + 14:table + 14 + 2 * segments]) + starts_at = table + 16 + 2 * segments + starts = struct.unpack(f">{segments}H", data[starts_at:starts_at + 2 * segments]) + deltas = struct.unpack(f">{segments}h", data[starts_at + 2 * segments:starts_at + 4 * segments]) + offsets_at = starts_at + 4 * segments + offsets = struct.unpack(f">{segments}H", data[offsets_at:offsets_at + 2 * segments]) + for segment, (start, end, delta, offset) in enumerate(zip(starts, ends, deltas, offsets)): + for code in range(start, min(end, 0xFFFE) + 1): + if offset == 0: + glyph = (code + delta) & 0xFFFF + else: + position = offsets_at + 2 * segment + offset + 2 * (code - start) + glyph = struct.unpack(">H", data[position:position + 2])[0] + glyph = (glyph + delta) & 0xFFFF if glyph else 0 + if glyph: + codepoints.add(code) + return codepoints + + +@lru_cache(maxsize=None) +def emoji_pattern(emoji_font: Path = EMOJI_FONT, text_font: Path = TEXT_FONT) -> re.Pattern: + """Emoji the text font leaves to the emoji font: a flag, or a base emoji with its variation selector, + skin tone and zero-width joins.""" + modifiers = {0xFE0F, 0x200D, 0x20E3, *range(0x1F3FB, 0x1F400), *range(0xE0020, 0xE0080)} + codes = sorted(font_codepoints(emoji_font) - font_codepoints(text_font) - modifiers) + ranges: list[list[int]] = [] + for code in codes: + if ranges and code == ranges[-1][1] + 1: + ranges[-1][1] = code + else: + ranges.append([code, code]) + base = "[" + "".join( + re.escape(chr(start)) if start == end else f"{re.escape(chr(start))}-{re.escape(chr(end))}" + for start, end in ranges + ) + "]" + modifier = "[️\U0001F3FB-\U0001F3FF]*" + return re.compile(f"[\U0001F1E6-\U0001F1FF]{{2}}|{base}{modifier}(?:‍{base}{modifier})*") + + +def typst_png(pages: list[str], targets: list[Path], typst: str, preamble: str, what: str) -> None: + """Render Typst markup to PNG, a page each, on white at 600 ppi, and save each page without alpha. + + Each page is as large as its content. A copy that would be wider than the + text column at 600 ppi is scaled down and declares a lower density, so it + keeps the size Typst prints it at. + """ + work = targets[0].parent + work.mkdir(parents=True, exist_ok=True) + source = work / "render.typ" + source.write_text( + "#set page(width: auto, height: auto, margin: 0pt, fill: white)\n" + preamble + + "\n#pagebreak()\n".join(pages) + "\n", + encoding="utf-8", + ) + run( + [ + # An unknown font is only a warning to Typst, so the fonts are found from this module, not the + # working directory. + typst, "compile", "--root", ".", "--ignore-system-fonts", + "--font-path", str(Path(__file__).resolve().parent / "pdf" / "fonts"), + "--format", "png", "--ppi", str(RENDER_PPI), str(source), str(work / "page-{p}.png"), + ], + what, + ) + for number, target in enumerate(targets, 1): + page = work / f"page-{number}.png" + with Image.open(page) as image: + flat = image.convert("RGB") + density = RENDER_PPI + if flat.width > MAX_RENDER_WIDTH: + density = RENDER_PPI * MAX_RENDER_WIDTH / flat.width + height = max(1, round(flat.height * MAX_RENDER_WIDTH / flat.width)) + flat = flat.resize((MAX_RENDER_WIDTH, height), Image.Resampling.LANCZOS) + flat.save(target, "PNG", dpi=(density, density)) + page.unlink() + + +def render_emoji(sequences: list[str], targets: list[Path], typst: str) -> None: + """Render emoji with Typst, which draws the COLRv1 glyphs: 10 pt, from the font's ascender to its descender.""" + typst_png( + sequences, targets, typst, + '#set text(font: "Noto Color Emoji", size: 10pt, top-edge: "ascender", bottom-edge: "descender")\n', + "typst (emoji)", + ) + + +def emoji_images( + doc: BeautifulSoup, emoji_dir: Path, typst: str, stats: Counter, render=render_emoji +) -> list[list[str]]: + """Render the colour emoji of the document as images, for the style to print instead (print edition). + + Typst writes colour emoji as a Type 3 font whose glyphs carry shadings and + soft masks: transparency in the book block, and content Ghostscript 10.08 + drops or crashes on when it converts the book to CMYK (P13). The style swaps + each sequence for its image with a show rule, so emoji in code and headings + print as images too. Returns (sequence, image) pairs, the longest sequence + first: the style gives the first pair precedence where sequences overlap. + """ + pattern = emoji_pattern() + sequences = { + match.group(0) + for node in doc.find_all(string=pattern) + if type(node) is NavigableString and not any(parent.name in ("script", "style") for parent in node.parents) + for match in pattern.finditer(node) + } + if not sequences: + return [] + ordered = sorted(sequences, key=lambda sequence: (-len(sequence), sequence)) + targets = [emoji_dir / ("-".join(f"{ord(char):x}" for char in sequence) + ".png") for sequence in ordered] + render(ordered, targets, typst) + stats["emoji rendered as images"] += len(ordered) + return [[sequence, typst_path(target)] for sequence, target in zip(ordered, targets)] + + +def keep_lead_with_heading(doc: BeautifulSoup, stats: Counter) -> None: + """Keep a short lead paragraph with its heading and what follows it (print edition). + + A heading is sticky, so it is never the last thing on a page - but a one-line + lead under it, such as the intended audience of a section, satisfies that and + the break falls after the line, leaving heading and line alone at the foot of + the page. A sticky lead carries both to where the section's content starts. + """ + for paragraph in doc.find_all("p"): + previous = paragraph.find_previous_sibling() + if previous is None or previous.name not in LEAD_HEADINGS: + continue + if len(paragraph.get_text(" ", strip=True)) > LEAD_MAX_CHARS: + continue + paragraph.wrap(doc.new_tag("div", attrs={"class": "keep-with-next"})) + stats["lead lines kept with their heading"] += 1 + + +def rasterize_transparent_svgs( + doc: BeautifulSoup, svg_dir: Path, typst: str, stats: Counter, render=typst_png +) -> None: + """Render SVG images that draw with transparency as PNG on white, at their natural size (print edition). + + Typst embeds such an SVG with transparency groups and soft masks, which + Ghostscript 10.08 draws incompletely when it writes PDF/X-4 (P13); telling it + to ignore transparency instead strikes arrows through the labels of an + Excalidraw diagram. Icons arrive as data URIs, other SVGs as files. + """ + found: list[tuple[Tag, Path]] = [] + for img in doc.find_all("img"): + src = img.get("src", "") + if src.startswith("data:image/svg+xml;base64,"): + markup = base64.b64decode(src.split(",", 1)[1]) + if not SVG_TRANSPARENCY.search(markup.decode("utf-8", errors="ignore")): + continue + source = svg_dir / f"{hashlib.sha256(markup).hexdigest()[:16]}.svg" + source.parent.mkdir(parents=True, exist_ok=True) + source.write_bytes(markup) + elif src.startswith("/") and src.lower().endswith(".svg"): + source = Path(src.lstrip("/")) + if not source.is_file() or not SVG_TRANSPARENCY.search(source.read_text(encoding="utf-8", errors="ignore")): + continue + else: + continue + found.append((img, source)) + if not found: + return + sources = sorted({source for _, source in found}) + targets = { + source: svg_dir / f"{source.stem}-{hashlib.sha256(str(source).encode()).hexdigest()[:8]}.png" + for source in sources + } + render( + [f'#image("{typst_path(source)}")' for source in sources], + [targets[source] for source in sources], + typst, "", "typst (SVG)", + ) + for img, source in found: + img["src"] = typst_path(targets[source]) + stats["SVG images with transparency rendered as PNG"] += 1 + + +def normalize( + doc: BeautifulSoup, + site_dir: Path, + print_images: Path | None = None, + low_resolution: dict[str, int] | None = None, +) -> tuple[Counter, list[str]]: """Rewrite the merged document into elements pandoc's HTML reader understands. Returns counts of what was rewritten and warnings about content the PDF @@ -1377,7 +1633,7 @@ def normalize(doc: BeautifulSoup, site_dir: Path, print_images: Path | None = No pair_tabs(doc, stats) mark_cards(doc, stats) inline_icons(doc, stats) - resolve_images(doc, site_dir, stats, warnings, print_images) + resolve_images(doc, site_dir, stats, warnings, print_images, low_resolution) embedded_media_to_links(doc, stats) settle_internal_links(doc, stats) return stats, warnings @@ -1493,12 +1749,33 @@ def unpadded_pages(compile_command: list[str], typ_path: Path) -> int: help="Which Typst binary typesets the PDF?", show_default=True, ) +@click.option( + "--normalize", "normalize_output", + is_flag=True, + envvar="PDF_NORMALIZE", + help="Also write a PDF/X-4 copy in CMYK with Ghostscript, and check that one? Print edition only.", +) +@click.option( + "--gray", "gray_output", + is_flag=True, + envvar="PDF_GRAY", + help="Also write a greyscale preview with Ghostscript, to see the black-and-white print on screen? " + "Print edition only.", +) def build_pdf( - build_version: str, edition_name: str, output_file: str | None, pandoc_binary: str, typst_binary: str + build_version: str, + edition_name: str, + output_file: str | None, + pandoc_binary: str, + typst_binary: str, + normalize_output: bool, + gray_output: bool, ) -> None: """Build a single PDF of the whole site with pandoc and Typst.""" version = build_version.strip() or "dev" print_edition = edition_name == "print" + if (normalize_output or gray_output) and not print_edition: + raise click.ClickException("--normalize and --gray belong to the print edition - add --edition print") suffix = "-print" if print_edition else "" out = Path(output_file or f"{DEFAULT_OUT_STEM}-{version.replace('.', '-')}{suffix}.pdf") # The print edition merges and normalizes differently, so its intermediate @@ -1549,12 +1826,29 @@ def build_pdf( print(f"Print edition: {count} {'part' if count == 1 else 'parts'} of {md} left out") print(f"Merged {pages} pages and {headings} section headings along {NAV_YML}") - stats, warnings = normalize(doc, SITE_DIR, work_dir / "images" if print_edition else None) + low_resolution: dict[str, int] = {} + stats, warnings = normalize(doc, SITE_DIR, work_dir / "images" if print_edition else None, low_resolution) + emoji: list[list[str]] = [] + if print_edition: + keep_lead_with_heading(doc, stats) + # Transparency Ghostscript cannot convert to PDF/X-4 becomes images (P13). + rasterize_transparent_svgs(doc, work_dir / "svg", typst, stats) + emoji = emoji_images(doc, work_dir / "emoji", typst, stats) for warning in warnings: print(f"WARNING: {warning}", file=sys.stderr) print("Normalized " + ", ".join(f"{count} {what}" for what, count in sorted(stats.items()))) work_dir.mkdir(parents=True, exist_ok=True) + if print_edition: + # The originals to replace or accept (backlog P15). + unaccepted = unaccepted_low_resolution(low_resolution, load_accepted_low_resolution()) + report = work_dir / "low-resolution.tsv" + report.write_text("ppi\timage\n" + "".join(f"{ppi}\t{image}\n" for image, ppi in unaccepted), encoding="utf-8") + if unaccepted: + print( + f"Print edition: {len(unaccepted)} images below {LOW_RESOLUTION_PPI} ppi at their printed size, " + f"listed in {report} - replace them, or accept them in {PRINT_YML}" + ) html_path = work_dir / "book.html" typ_path = work_dir / "book.typ" html_path.write_text(str(doc), encoding="utf-8") @@ -1589,6 +1883,7 @@ def build_pdf( file=sys.stderr, ) inputs["authors"] = ", ".join(names) + inputs["emoji"] = json.dumps(emoji) for key, value in inputs.items(): command += ["--input", f"{key}={value}"] if print_edition and unpadded_pages(command, typ_path) % 2: @@ -1597,3 +1892,16 @@ def build_pdf( run(command + [str(typ_path), str(out)], "typst") print(f"PDF written to {out} ({out.stat().st_size // 1024} KB)") + + # The book block is checked before Ghostscript runs for minutes on it, and the + # PDF/X-4 copy again for what the conversion must deliver. + if print_edition and not print_report(out, run_preflight(out)): + raise click.ClickException(f"preflight found errors in {out} - it is written, but not ready for print") + if normalize_output: + final = normalize_pdf(out, normalized_path(out), ensure_profile(), title=f"{BOOK_TITLE} - {BOOK_CONTEXT}") + print(f"PDF/X-4 written to {final} ({final.stat().st_size // 1024} KB)") + if not print_report(final, run_preflight(final, pdfx=True)): + raise click.ClickException(f"preflight found errors in {final} - it is written, but not ready for print") + if gray_output: + preview = gray_preview(out, gray_path(out), title=f"{BOOK_TITLE} - {BOOK_CONTEXT}") + print(f"Greyscale preview written to {preview} ({preview.stat().st_size // 1024} KB)") diff --git a/tools/image_widths.py b/tools/image_widths.py new file mode 100644 index 000000000..76ba6a43e --- /dev/null +++ b/tools/image_widths.py @@ -0,0 +1,133 @@ +"""Write the missing width of raster images into the Markdown sources. + +An image without a width prints at the pixel density it declares - 72 dpi when +it declares none - and the site shows it at its pixel size. `dec-tool +image-widths` writes `{ width="NN%" }` instead: the image's natural width as a +share of the full page width. A screenshot then prints at the size it had on the +screen it was captured on, and the site shows it in the same proportion +(tasks/spec.md, §11, D18). Generated pages are left alone: their generators own +them. + +Without `--fix` the command lists the images without a width and fails, so a +check can catch a new screenshot; with `--fix` it writes the widths. +""" +from __future__ import annotations + +import math +import re +import urllib.parse +from pathlib import Path + +import click +from PIL import Image, UnidentifiedImageError + +DOCS_DIR = Path("docs") +# The full page width in CSS pixels: the 16 cm text column of the print edition, at 96 per inch. +FULL_PAGE_WIDTH = 16 / 2.54 * 96 +RASTER_SUFFIXES = {".png", ".jpg", ".jpeg", ".gif", ".webp"} +# The marker a generated page carries (.claude/docs-guidelines/repo-conventions.md). +GENERATED_MARK = re.compile(r"generated - do not change it manually|auto-generated", re.IGNORECASE) +# A Markdown image, with an optional title, and the attribute list that may follow it. +IMAGE = re.compile(r'!\[[^\]]*\]\((?P[^)\s]+)(?:\s+"[^"]*")?\)(?P\{[^}\n]*\})?') +FENCE = re.compile(r"^\s*(```|~~~)") + + +def capture_scale(dpi: float | None) -> float: + """How many image pixels one CSS pixel took on the screen the image was captured on. + + 144 dpi is a Retina capture on macOS, twice its 72 dpi base. Any other declared + density counts against 96 dpi. An image that declares none, or less than 96 dpi, + has scale 1. + """ + if not dpi or dpi < 96: + return 1.0 + if round(dpi) == 144: + return 2.0 + return dpi / 96 + + +def width_percent(pixels: int, dpi: float | None) -> int: + """The image's natural width as a share of the full page width, rounded to 5 %, between 5 and 100 %.""" + natural = pixels / capture_scale(dpi) + percent = 5 * math.floor(natural / FULL_PAGE_WIDTH * 20 + 0.5) + return max(5, min(100, percent)) + + +def image_width(page_dir: Path, source: str) -> int | None: + """The width to write for an image a page references, or None for one that gets none. + + Remote images, SVGs and files that are missing or unreadable get none. + """ + if re.match(r"^[a-z][a-z0-9+.-]*:", source, re.IGNORECASE) or source.startswith("/"): + return None + path = page_dir / urllib.parse.unquote(source.split("#")[0].split("?")[0]) + if path.suffix.lower() not in RASTER_SUFFIXES or not path.is_file(): + return None + try: + with Image.open(path) as image: + dpi = image.info.get("dpi") + return width_percent(image.width, float(dpi[0]) if dpi and dpi[0] else None) + except (OSError, UnidentifiedImageError): + return None + + +def add_widths(text: str, page_dir: Path) -> tuple[str, list[tuple[str, int]]]: + """A page's Markdown with a width on every raster image that lacks one, and the images that got one. + + An existing attribute list gains the width; an image without one gets + `{ width="NN%" }`. An image that would get 100 % gets none: it fills the + column in print either way, and the width would only stretch a narrower + screenshot to the article width on the site. Images inside fenced code + blocks stay as they are. + """ + lines = text.split("\n") + changes: list[tuple[str, int]] = [] + fenced = False + + def widen(match: re.Match) -> str: + attrs = match.group("attrs") or "" + if "width" in attrs: + return match.group(0) + percent = image_width(page_dir, match.group("src")) + if percent is None or percent >= 100: + return match.group(0) + changes.append((match.group("src"), percent)) + if attrs: + return match.group(0)[: -len(attrs)] + attrs[:-1].rstrip() + f' width="{percent}%" }}' + return match.group(0) + f'{{ width="{percent}%" }}' + + for index, line in enumerate(lines): + if FENCE.match(line): + fenced = not fenced + continue + if not fenced and "![" in line: + lines[index] = IMAGE.sub(widen, line) + return "\n".join(lines), changes + + +@click.command(name="image-widths") +@click.option("--fix", is_flag=True, help="Write the missing widths into the pages instead of listing them.") +@click.argument("paths", nargs=-1, type=click.Path(exists=True, path_type=Path)) +def image_widths(fix: bool, paths: tuple[Path, ...]) -> None: + """List the raster images without a width in pages that are not generated - or write their widths.""" + roots = paths or (DOCS_DIR,) + pages = sorted({page for root in roots for page in ([root] if root.is_file() else root.rglob("*.md"))}) + count = 0 + for page in pages: + text = page.read_text(encoding="utf-8") + if GENERATED_MARK.search("\n".join(text.splitlines()[:15])): + continue + result, changes = add_widths(text, page.parent) + if not changes: + continue + count += len(changes) + for source, percent in changes: + print(f'{page}: {source} width="{percent}%"') + if fix: + page.write_text(result, encoding="utf-8") + if fix: + print(f"{count} image widths written") + elif count: + raise click.ClickException(f"{count} images have no width - run `dec-tool image-widths --fix`") + else: + print("no image needs a width") diff --git a/tools/pdf/filter.lua b/tools/pdf/filter.lua index 44e3d2f11..e01305977 100644 --- a/tools/pdf/filter.lua +++ b/tools/pdf/filter.lua @@ -41,6 +41,10 @@ function Div(el) return raw('#chapter-break()') elseif c:includes('part-contents') then return raw('#part-contents()') + elseif c:includes('keep-with-next') then + -- A short lead line under a heading: it stays with the block after it, so + -- heading and line do not sit alone at the foot of a page. + return wrap('#keep-with-next[', el.content, ']') elseif c:includes('operator-fields') then return wrap('#operator-fields[', el.content, ']') elseif c:includes('part-end') then diff --git a/tools/pdf/print.yml b/tools/pdf/print.yml index 002eaaa0f..36e025842 100644 --- a/tools/pdf/print.yml +++ b/tools/pdf/print.yml @@ -50,6 +50,11 @@ authors: # darker in black and white. body-weight: light +# Images below 150 ppi at their printed size that are accepted as they are, by +# their path under docs/. The print build lists the others in +# dist/pdf/print/low-resolution.tsv, to replace (backlog P15). +accepted-low-resolution: [] + # How a navigation section or a page is printed, keyed by the docs/ directory # its pages live in (ending in /) or by a single page (ending in .md). What is # not listed prints in full; keys do not nest. diff --git a/tools/pdf/style.typ b/tools/pdf/style.typ index cea02403a..cbeed3274 100644 --- a/tools/pdf/style.typ +++ b/tools/pdf/style.typ @@ -393,6 +393,12 @@ // ----------------------------------------------------------------------------- // The book: title page, contents, then the documentation itself. // ----------------------------------------------------------------------------- +// DEVIATION (print): a heading is sticky, so it never ends a page alone - but a +// one-line lead under it, such as the intended audience, satisfies that and +// leaves the two of them at the foot of the page. The build wraps such a lead +// (build_pdf.py, keep_lead_with_heading), and this carries it to the next block. +#let keep-with-next(body) = block(sticky: true, body) + #let book( title: "", subtitle: "", @@ -489,6 +495,17 @@ // built into Typst instead of printing as an empty box. fallback: true, ) + // DEVIATION (print): colour emoji print as images the build renders + // (build_pdf.py, emoji_images) - Typst writes their glyphs as a Type 3 font + // with shadings and soft masks, which Ghostscript cannot convert to PDF/X-4. + // Noto Color Emoji spans 950 + 250 of 1024 units from ascender to descender: + // the image is that high and lowered by the descender, where the glyph sits. + // The first pair's rule is the innermost, so it wins where sequences overlap. + let emoji-images = json(bytes(sys.inputs.at("emoji", default: "[]"))) + show: body => emoji-images.fold(body, (inner, pair) => { + show pair.at(0): box(baseline: 0.244em, image(pair.at(1), height: 1.172em, alt: pair.at(0))) + inner + }) show heading: set text(hyphenate: false) // DEVIATION (print): the house spacing suits a two-page policy; over a // thousand printed pages it costs a tenth of the book (tasks/spec.md, §5). diff --git a/tools/pdf_normalize.py b/tools/pdf_normalize.py new file mode 100644 index 000000000..930e5d092 --- /dev/null +++ b/tools/pdf_normalize.py @@ -0,0 +1,317 @@ +"""Turn the print edition into PDF/X-4 in CMYK with Ghostscript (tasks/spec.md, §7). + +The recipe was verified with Ghostscript 10.08: `-dPDFX=4` with a prefix file +that writes the PDF/X-4 marker and the output intent, CMYK conversion, images +downsampled to 300 ppi and re-encoded losslessly, and annotations dropped. +Ghostscript gives three traps a wide berth only if they are known: a page with +an annotation makes it write plain PDF with a one-line warning, `-dPDFX=4` alone +writes no marker, and `-sOutputICCProfile` together with `-dPDFX` crashes it. So +the profile goes in through the prefix file only, and the output is checked for +the marker instead of trusting the exit code. + +The output intent is ISO Coated v2 (ECI), the FOGRA39 profile BoD assumes for +CMYK. Its copyright does not allow redistribution, so it is not committed: it is +fetched from the ECI once, checked against its checksum and cached in the +gitignored `dist/icc/`, unless `--icc-profile` or PDF_ICC_PROFILE names a copy. + +`--gray` runs the same pass in greyscale instead: a preview of how the book +prints in black and white, for checking the palette on screen (P11). It is +PDF/X-4 as well, with the generic grey profile Ghostscript ships as its output +intent - one component instead of four, and no colour registry to name. +""" +from __future__ import annotations + +import glob +import hashlib +import io +import os +import shutil +import subprocess +import tempfile +import time +import urllib.error +import urllib.request +import zipfile +from pathlib import Path + +import click +from pypdf import PdfReader + +PROFILE_URL = "https://www.eci.org/lib/exe/eci_offset_2009.zip" +PROFILE_MEMBER = "ECI_Offset_2009/ISOcoated_v2_eci.icc" +PROFILE_SHA256 = "128dc02f7246cc3807af0323695379f64151a8f27a587736acc59f8b6ce894b8" +PROFILE_CACHE = Path("dist/icc/ISOcoated_v2_eci.icc") +FOGRA39 = "FOGRA39" +FOGRA39_INFO = "ISO Coated v2 (ECI)" +COLOR_REGISTRY = "http://www.color.org" +# The greyscale preview prints by the generic grey profile Ghostscript ships. +GRAY_PROFILE_PATTERNS = ( + "/opt/homebrew/share/ghostscript/*/iccprofiles/default_gray.icc", + "/opt/homebrew/Cellar/ghostscript/*/share/ghostscript/iccprofiles/default_gray.icc", + "/usr/local/share/ghostscript/*/iccprofiles/default_gray.icc", + "/usr/share/ghostscript/*/iccprofiles/default_gray.icc", + "/usr/share/color/icc/ghostscript/default_gray.icc", +) +GRAY_CONDITION = "sGray" +GRAY_CONDITION_INFO = "Artifex Software sGray ICC Profile (Ghostscript default_gray.icc)" +# Both passes: images downsampled to 300 ppi and re-encoded losslessly, annotations dropped. +IMAGE_AND_ANNOTATION_OPTIONS = [ + "-dDownsampleColorImages=true", "-dColorImageDownsampleType=/Bicubic", + "-dColorImageResolution=300", "-dColorImageDownsampleThreshold=1.0", + "-dDownsampleGrayImages=true", "-dGrayImageDownsampleType=/Bicubic", + "-dGrayImageResolution=300", "-dGrayImageDownsampleThreshold=1.0", + "-dAutoFilterColorImages=false", "-dColorImageFilter=/FlateEncode", + "-dAutoFilterGrayImages=false", "-dGrayImageFilter=/FlateEncode", + "-dPreserveAnnots=false", +] + + +def postscript_string(text: str) -> str: + """A PostScript string literal: backslashes and parentheses escaped.""" + escaped = text.replace("\\", "\\\\").replace("(", "\\(").replace(")", "\\)") + return f"({escaped})" + + +def pdfx_prefix( + profile: Path, + title: str, + components: int = 4, + identifier: str = FOGRA39, + condition: str = FOGRA39_INFO, + registry: str | None = COLOR_REGISTRY, +) -> str: + """The PostScript Ghostscript reads before the PDF: the PDF/X-4 marker, the title and the output intent. + + The book block prints by FOGRA39 in four components; the greyscale preview by + the grey profile in one, and an unregistered condition names no registry. + """ + intent = [ + "[{OutputIntent_PDFX} <<", + " /Type /OutputIntent", + " /S /GTS_PDFX", + f" /OutputCondition {postscript_string(condition)}", + f" /OutputConditionIdentifier {postscript_string(identifier)}", + ] + if registry: + intent.append(f" /RegistryName {postscript_string(registry)}") + return "\n".join([ + "%!", + "% PDF/X-4 definitions of dec-tool pdf-normalize (tasks/spec.md, section 7).", + f"[/GTS_PDFXVersion (PDF/X-4) /Title {postscript_string(title)} /Trapped /False /DOCINFO pdfmark", + "[/_objdef {icc_PDFX} /type /stream /OBJ pdfmark", + f"[{{icc_PDFX}} <> /PUT pdfmark", + f"[{{icc_PDFX}} {postscript_string(str(profile))} (r) file /PUT pdfmark", + "[/_objdef {OutputIntent_PDFX} /type /dict /OBJ pdfmark", + *intent, + f" /Info {postscript_string(condition)}", + " /DestOutputProfile {icc_PDFX}", + ">> /PUT pdfmark", + "[{Catalog} <> /PUT pdfmark", + "", + ]) + + +def ghostscript_command(ghostscript: str, prefix: Path, profile: Path, source: Path, output: Path) -> list[str]: + """The Ghostscript call of tasks/spec.md, §7: PDF/X-4, CMYK, images at 300 ppi, no annotations.""" + return [ + ghostscript, + "-dPDFX=4", "-dBATCH", "-dNOPAUSE", + f"--permit-file-read={profile}", + "-sDEVICE=pdfwrite", + "-sColorConversionStrategy=CMYK", "-sProcessColorModel=DeviceCMYK", + *IMAGE_AND_ANNOTATION_OPTIONS, + f"-sOutputFile={output}", + str(prefix), str(source), + ] + + +def gray_command(ghostscript: str, prefix: Path, profile: Path, source: Path, output: Path) -> list[str]: + """The greyscale preview: the same pass in DeviceGray, PDF/X-4 by the grey profile.""" + return [ + ghostscript, + "-dPDFX=4", "-dBATCH", "-dNOPAUSE", + f"--permit-file-read={profile}", + "-sDEVICE=pdfwrite", + "-sColorConversionStrategy=Gray", "-sProcessColorModel=DeviceGray", + *IMAGE_AND_ANNOTATION_OPTIONS, + f"-sOutputFile={output}", + str(prefix), str(source), + ] + + +def is_pdfx4(path: Path) -> bool: + """Whether a PDF declares PDF/X-4 and carries an output intent.""" + reader = PdfReader(path) + metadata = reader.metadata or {} + return metadata.get("/GTS_PDFXVersion") == "PDF/X-4" and "/OutputIntents" in reader.trailer["/Root"] + + +def fetch_profile(target: Path, url: str = PROFILE_URL, member: str = PROFILE_MEMBER, sha256: str = PROFILE_SHA256) -> Path: + """Download the profile's archive, take the profile out, check its checksum and store it at `target`.""" + print(f"Fetching the ICC profile from {url}", flush=True) + try: + with urllib.request.urlopen(url, timeout=120) as response: + archive = response.read() + with zipfile.ZipFile(io.BytesIO(archive)) as bundle: + data = bundle.read(member) + except (urllib.error.URLError, zipfile.BadZipFile, KeyError, OSError) as error: + raise click.ClickException(f"cannot fetch {member} from {url} ({error})") from error + if hashlib.sha256(data).hexdigest() != sha256: + raise click.ClickException(f"{member} from {url} does not match its checksum - not using it") + target.parent.mkdir(parents=True, exist_ok=True) + target.write_bytes(data) + return target + + +def ensure_profile(profile: Path | None = None) -> Path: + """The output intent profile: the one named, else PDF_ICC_PROFILE, else the cached ECI profile, fetched if missing.""" + if profile is None and os.environ.get("PDF_ICC_PROFILE"): + profile = Path(os.environ["PDF_ICC_PROFILE"]) + if profile is not None: + if not profile.is_file(): + raise click.ClickException(f"ICC profile {profile} not found") + return profile + if PROFILE_CACHE.is_file() and hashlib.sha256(PROFILE_CACHE.read_bytes()).hexdigest() == PROFILE_SHA256: + return PROFILE_CACHE + return fetch_profile(PROFILE_CACHE) + + +def ensure_gray_profile(profile: Path | None = None) -> Path: + """The grey output intent profile: the one named, else the generic one Ghostscript ships.""" + if profile is not None: + if not profile.is_file(): + raise click.ClickException(f"ICC profile {profile} not found") + return profile + found = [path for pattern in GRAY_PROFILE_PATTERNS for path in sorted(glob.glob(pattern))] + if not found: + raise click.ClickException( + "no grey ICC profile found - Ghostscript ships default_gray.icc with its iccprofiles; " + "name one with --icc-profile" + ) + return Path(found[0]) + + +def normalized_path(pdf: Path) -> Path: + """Where the PDF/X-4 copy of a PDF goes: next to it, with -x4 before the suffix.""" + return pdf.with_name(f"{pdf.stem}-x4.pdf") + + +def gray_path(pdf: Path) -> Path: + """Where the greyscale preview of a PDF goes: next to it, with -gray-x4 before the suffix - it is PDF/X-4 too.""" + return pdf.with_name(f"{pdf.stem}-gray-x4.pdf") + + +def ghostscript_binary(ghostscript: str) -> str: + binary = shutil.which(ghostscript) + if binary is None: + raise click.ClickException(f"{ghostscript} not found - install Ghostscript or name the binary with --ghostscript") + return binary + + +def run_ghostscript(command: list[str], output: Path) -> subprocess.CompletedProcess: + """Run Ghostscript, and fail - leaving no output - when it exits with an error or leaves content out.""" + started = time.monotonic() + result = subprocess.run(command, capture_output=True, text=True, check=False) + if result.returncode != 0: + output.unlink(missing_ok=True) + lines = (result.stdout + result.stderr).splitlines() + pages = [line for line in lines if line.startswith("Page ")] + errors = [line.strip() for line in lines if "error" in line.lower()] + where = f" after {pages[-1].lower()}" if pages else "" + raise click.ClickException( + f"Ghostscript failed{where} (exit {result.returncode}), no file written:\n" + "\n".join(errors[:20]) + ) + if "error executing PDF token" in result.stdout + result.stderr: + # Ghostscript "repairs" such an error by leaving out what it could not + # interpret - a colour emoji glyph, for one. + output.unlink(missing_ok=True) + raise click.ClickException( + "Ghostscript left out content it could not interpret (error executing PDF token), no file written - " + "look for Type 3 fonts in the source with `dec-tool pdf-preflight`" + ) + print(f"Ghostscript took {time.monotonic() - started:.0f}s") + return result + + +def normalize(source: Path, output: Path, profile: Path, ghostscript: str = "gs", title: str = "") -> Path: + """Write `source` as PDF/X-4 in CMYK to `output`, and fail if Ghostscript fell back to plain PDF.""" + binary = ghostscript_binary(ghostscript) + title = title or (PdfReader(source).metadata or {}).get("/Title") or source.stem + with tempfile.TemporaryDirectory() as work: + prefix = Path(work) / "pdfx.ps" + prefix.write_text(pdfx_prefix(profile.resolve(), str(title)), encoding="utf-8") + result = run_ghostscript(ghostscript_command(binary, prefix, profile.resolve(), source, output), output) + require_pdfx4(output, result) + return output + + +def require_pdfx4(output: Path, result: subprocess.CompletedProcess) -> None: + """Fail if Ghostscript wrote plain PDF instead: one annotation left anywhere makes it fall back.""" + if is_pdfx4(output): + return + warnings = "\n".join(line for line in result.stdout.splitlines() if "warning" in line.lower())[-800:] + raise click.ClickException( + f"Ghostscript wrote plain PDF instead of PDF/X-4 to {output} - a page with an annotation makes it fall " + f"back. Its warnings:\n{warnings}" + ) + + +def gray_preview( + source: Path, output: Path, profile: Path | None = None, ghostscript: str = "gs", title: str = "" +) -> Path: + """Write `source` in greyscale to `output` as PDF/X-4: how the book block prints in black and white (P11).""" + binary = ghostscript_binary(ghostscript) + profile = ensure_gray_profile(profile) + title = title or (PdfReader(source).metadata or {}).get("/Title") or source.stem + with tempfile.TemporaryDirectory() as work: + prefix = Path(work) / "pdfx-gray.ps" + prefix.write_text( + pdfx_prefix( + profile.resolve(), str(title), + components=1, identifier=GRAY_CONDITION, condition=GRAY_CONDITION_INFO, registry=None, + ), + encoding="utf-8", + ) + result = run_ghostscript(gray_command(binary, prefix, profile.resolve(), source, output), output) + require_pdfx4(output, result) + return output + + +@click.command(name="pdf-normalize") +@click.argument("pdf", type=click.Path(exists=True, dir_okay=False, path_type=Path)) +@click.option( + "--output-file", "-o", + type=click.Path(dir_okay=False, path_type=Path), + default=None, + help="Where to write the file? [default: next to the PDF, with -x4, or -gray-x4 with --gray]", +) +@click.option( + "--gray", + is_flag=True, + help="Write a greyscale preview instead: how the book prints in black and white, PDF/X-4 by a grey profile.", +) +@click.option( + "--icc-profile", + type=click.Path(exists=True, dir_okay=False, path_type=Path), + envvar="PDF_ICC_PROFILE", + default=None, + help="Which profile is the output intent? [default: ISO Coated v2 (ECI), fetched once into dist/icc/; " + "with --gray, the grey profile Ghostscript ships]", +) +@click.option( + "--ghostscript", + default="gs", + envvar="GHOSTSCRIPT", + show_default=True, + help="Which Ghostscript binary converts the PDF?", +) +def pdf_normalize(pdf: Path, output_file: Path | None, gray: bool, icc_profile: Path | None, ghostscript: str) -> None: + """Write a PDF as PDF/X-4 in CMYK with Ghostscript, for print on demand - or in greyscale, as a preview.""" + if gray: + output = output_file or gray_path(pdf) + gray_preview(pdf, output, icc_profile, ghostscript) + print(f"Greyscale preview written to {output} ({output.stat().st_size // 1024} KB)") + return + output = output_file or normalized_path(pdf) + normalize(pdf, output, ensure_profile(icc_profile), ghostscript) + print(f"PDF/X-4 written to {output} ({output.stat().st_size // 1024} KB)") diff --git a/tools/pdf_preflight.py b/tools/pdf_preflight.py new file mode 100644 index 000000000..2e65a28e7 --- /dev/null +++ b/tools/pdf_preflight.py @@ -0,0 +1,425 @@ +"""Check the book block of the print edition before it goes to print (tasks/spec.md, §8; backlog P14). + +The checks are what BoD rejects or prints badly: pages that are not A4, an odd +page count or too many pages, fonts that are not embedded, images below 300 ppi, +soft masks, annotations, page numbers on the inner edge, page furniture on blank +pages, and - as a warning - grey areas lighter than 20 % black. With `--pdfx` +the file must also declare PDF/X-4 with a FOGRA39 output intent and hold only +CMYK or grey images of at most 300 ppi. + +The page data comes from pypdf and from poppler's pdffonts, pdfimages and +pdftotext. +""" +from __future__ import annotations + +import html +import re +import shutil +import subprocess +from dataclasses import dataclass, field +from pathlib import Path + +import click +from pypdf import PdfReader + +from tools.pdf_normalize import is_pdfx4 + +A4 = (595.28, 841.89) +# The output intent of the book block; the greyscale preview names its grey profile instead. +FOGRA39 = "FOGRA39" +SIZE_TOLERANCE = 1.0 +MAX_PAGES = 1200 +MIN_PPI = 300 +# pdfimages rounds densities to whole pixels. +PPI_TOLERANCE = 1 +MIN_AREA_BLACK = 0.20 +# The share of the page height below which the footer sits. +FOOTER_BAND = 0.9 +# The operators that paint an area, and those that paint anything else. +AREA_PAINTING = {b"f", b"F", b"f*", b"B", b"B*", b"b", b"b*"} +OTHER_PAINTING = {b"S", b"s", b"Do", b"sh", b"BI"} +TOKEN = re.compile( + rb"\((?:\\.|[^\\()])*\)" # a literal string + rb"|<<|>>|<[0-9A-Fa-f\s]*>" # dictionary delimiters and hex strings + rb"|\[|\]" + rb"|/[^\s/\[\]()<>{}%]*" # a name + rb"|[-+]?(?:\d+\.?\d*|\.\d+)" # a number + rb"|[A-Za-z'\"*]+" # an operator +) +WORD = re.compile(r'([^<]*)') + + +@dataclass +class Check: + """One preflight check: its findings, and whether they are errors or warnings.""" + + name: str + problems: list[str] = field(default_factory=list) + level: str = "error" + + @property + def passed(self) -> bool: + return not self.problems + + +def check_page_count(count: int, max_pages: int = MAX_PAGES) -> Check: + check = Check("even page count within the limit") + if count % 2: + check.problems.append(f"{count} pages - a printed book needs an even count") + if count > max_pages: + check.problems.append(f"{count} pages - more than {max_pages}") + return check + + +def check_page_sizes(sizes: list[tuple[float, float]]) -> Check: + check = Check("A4 pages") + for number, (width, height) in enumerate(sizes, 1): + if abs(width - A4[0]) > SIZE_TOLERANCE or abs(height - A4[1]) > SIZE_TOLERANCE: + check.problems.append(f"page {number}: {width:g} x {height:g} pt, not A4") + return check + + +def parse_pdffonts(text: str) -> list[dict]: + """The rows of `pdffonts`: name, type, encoding and whether the font is embedded.""" + rows = [] + for line in text.splitlines()[2:]: + tokens = line.split() + if len(tokens) < 7: + continue + # The name is one token, the type may be several, the last six are fixed columns. + rows.append({ + "name": tokens[0], + "type": " ".join(tokens[1:-6]), + "encoding": tokens[-6], + "emb": tokens[-5], + }) + return rows + + +def check_fonts(rows: list[dict]) -> Check: + """Fonts are embedded, and none is a Type 3 font. + + Typst writes colour emoji as a Type 3 font whose glyphs carry shadings and + soft masks - transparency, and content Ghostscript 10.08 drops or crashes on + when it converts to CMYK. The print build renders them as images instead. + """ + check = Check("fonts embedded, no Type 3 fonts") + for row in rows: + if row["emb"] != "yes": + check.problems.append(f"{row['name']} ({row['type']}) is not embedded") + if row["type"] == "Type 3": + check.problems.append(f"{row['name']} is a Type 3 font - colour glyphs the print build did not render as images") + return check + + +def parse_pdfimages(text: str) -> list[dict]: + """The rows of `pdfimages -list`.""" + rows = [] + for line in text.splitlines(): + tokens = line.split() + if len(tokens) != 16 or not tokens[0].isdigit(): + continue + + def number(value: str) -> int: + return int(value) if value.isdigit() else 0 + + rows.append({ + "page": int(tokens[0]), + "num": int(tokens[1]), + "type": tokens[2], + "width": number(tokens[3]), + "height": number(tokens[4]), + "color": tokens[5], + "x_ppi": number(tokens[12]), + "y_ppi": number(tokens[13]), + }) + return rows + + +def check_images(rows: list[dict], pdfx: bool = False) -> Check: + check = Check("images at 300 ppi" + (", CMYK and at most 300 ppi" if pdfx else "")) + for row in rows: + if row["type"] != "image": + continue + ppi = min(row["x_ppi"], row["y_ppi"]) + # A small image is a whole pixel away from any density: 49 pixels across an + # emoji are 294 ppi where 50 would be 300. Ghostscript's downsampling rounds + # to whole pixels, so one pixel either way passes. + pixels = max(1, min(row["width"], row["height"])) + where = f"page {row['page']}: image {row['num']}" + if ppi and ppi * (pixels + 1) / pixels < MIN_PPI - PPI_TOLERANCE: + check.problems.append(f"{where} at {ppi} ppi") + if pdfx and row["color"] not in ("cmyk", "gray"): + check.problems.append(f"{where} in {row['color']}, not CMYK") + if pdfx and pixels > 1 and ppi * (pixels - 1) / pixels > MIN_PPI + PPI_TOLERANCE: + check.problems.append(f"{where} at {ppi} ppi") + return check + + +def check_soft_masks(rows: list[dict]) -> Check: + check = Check("no soft masks") + for row in rows: + if row["type"] == "smask": + check.problems.append(f"page {row['page']}: soft mask on image {row['num']}") + return check + + +def resource_transparency(resources, found: set[str], depth: int = 0) -> None: + """Collect the transparency a page's resources draw with, down through forms, patterns and Type 3 fonts.""" + if resources is None or depth > 8: + return + resources = resources.get_object() + for state in resources.get("/ExtGState", {}).values(): + state = state.get_object() + if state.get("/SMask", "/None") != "/None": + found.add("soft mask") + if float(state.get("/ca", 1)) < 1 or float(state.get("/CA", 1)) < 1: + found.add("alpha") + for xobject in resources.get("/XObject", {}).values(): + xobject = xobject.get_object() + if xobject.get("/Subtype") == "/Form": + if "/Group" in xobject: + found.add("transparency group") + resource_transparency(xobject.get("/Resources"), found, depth + 1) + for pattern in resources.get("/Pattern", {}).values(): + resource_transparency(pattern.get_object().get("/Resources"), found, depth + 1) + for font in resources.get("/Font", {}).values(): + font = font.get_object() + if font.get("/Subtype") == "/Type3": + resource_transparency(font.get("/Resources"), found, depth + 1) + + +def read_transparency(path: Path) -> dict[int, set[str]]: + """The kinds of transparency each page draws with, by page number.""" + found: dict[int, set[str]] = {} + for number, page in enumerate(PdfReader(path).pages, 1): + found[number] = set() + resource_transparency(page.get("/Resources"), found[number]) + return found + + +def check_transparency(found: dict[int, set[str]]) -> Check: + """No transparency: BoD asks for none, and Ghostscript draws it incompletely when it writes PDF/X-4.""" + check = Check("no transparency") + for number, kinds in sorted(found.items()): + if kinds: + check.problems.append(f"page {number}: {', '.join(sorted(kinds))}") + return check + + +def check_annotations(counts: list[int]) -> Check: + check = Check("no annotations") + for number, count in enumerate(counts, 1): + if count: + check.problems.append(f"page {number}: {count} annotations") + return check + + +def check_pdfx(path: Path, condition: str = FOGRA39) -> Check: + """PDF/X-4 with the output intent the edition prints by: FOGRA39 for the book block in CMYK, + the grey profile's condition for the greyscale preview.""" + check = Check(f"PDF/X-4 with a {condition} output intent") + if not is_pdfx4(path): + check.problems.append("no PDF/X-4 marker or no output intent") + return check + intents = PdfReader(path).trailer["/Root"]["/OutputIntents"] + identifiers = [str(intent.get_object().get("/OutputConditionIdentifier", "")) for intent in intents] + if not any(condition in identifier for identifier in identifiers): + check.problems.append(f"output intent {', '.join(identifiers) or 'without an identifier'}, not {condition}") + return check + + +def check_page_numbers(words: dict[int, list[tuple]], sizes: dict[int, tuple[float, float]]) -> Check: + """A page number sits at the outer edge: left on an even page, right on an odd one.""" + check = Check("page numbers on the outer edge") + for number, page_words in sorted(words.items()): + width, height = sizes[number] + folios = [word for word in page_words if word[0] == str(number) and word[2] > height * FOOTER_BAND] + if not folios: + continue + _, x_min, _, x_max, _ = folios[-1] + outer = x_max < width / 2 if number % 2 == 0 else x_min > width / 2 + if not outer: + check.problems.append(f"page {number}: page number on the inner edge") + return check + + +def check_blank_pages( + words: dict[int, list[tuple]], sizes: dict[int, tuple[float, float]], painted: dict[int, bool] +) -> Check: + """A page with nothing on it but its footer is a blank page that still carries page furniture.""" + check = Check("blank pages carry no page furniture") + for number, page_words in sorted(words.items()): + _, height = sizes[number] + if page_words and not painted.get(number, False) and all(word[2] > height * FOOTER_BAND for word in page_words): + check.problems.append(f"page {number}: only page furniture") + return check + + +def area_tints(content: bytes) -> tuple[set[float], bool]: + """The tints lighter than 20 % black that a page's content fills areas with, and whether it paints at all. + + Follows the fill colour through the colour operators and the graphics state; + text in string literals is skipped. + """ + operands: list[float] = [] + tint = 1.0 + saved: list[float] = [] + light: set[float] = set() + painted = False + + def black(values: list[float]) -> float: + if len(values) == 1: + return 1 - values[0] + if len(values) == 3: + red, green, blue = values + return 1 - (0.299 * red + 0.587 * green + 0.114 * blue) + cyan, magenta, yellow, key = values + return 1 - (1 - key) * (1 - (0.3 * cyan + 0.59 * magenta + 0.11 * yellow)) + + for match in TOKEN.finditer(content): + token = match.group(0) + head = token[:1] + if head in b"-+.0123456789": + try: + operands.append(float(token)) + continue + except ValueError: + pass + if head in b"(<[]/" or token == b">>": + continue + if token == b"q": + saved.append(tint) + elif token == b"Q": + tint = saved.pop() if saved else 1.0 + elif token == b"g" and operands: + tint = black(operands[-1:]) + elif token == b"rg" and len(operands) >= 3: + tint = black(operands[-3:]) + elif token == b"k" and len(operands) >= 4: + tint = black(operands[-4:]) + elif token in (b"sc", b"scn") and len(operands) in (1, 3, 4): + tint = black(operands) + elif token == b"cs": + tint = 1.0 + elif token in AREA_PAINTING: + painted = True + # 0.8 g is 20 % black, 0.19999... in floating point. + if 0.005 < tint < MIN_AREA_BLACK - 0.005: + light.add(round(tint, 2)) + elif token in OTHER_PAINTING: + painted = True + operands = [] + return light, painted + + +def check_light_areas(tints: dict[int, set[float]]) -> Check: + """Grey areas lighter than 20 % black print unevenly - a warning, as artwork and emoji carry them too.""" + pages_by_tint: dict[float, list[int]] = {} + for number, page_tints in sorted(tints.items()): + for tint in page_tints: + pages_by_tint.setdefault(tint, []).append(number) + check = Check("grey areas at least 20 % black", level="warning") + for tint, pages in sorted(pages_by_tint.items()): + label = "page" if len(pages) == 1 else "pages" + shown = ", ".join(str(page) for page in pages[:8]) + more = f" and {len(pages) - 8} more" if len(pages) > 8 else "" + check.problems.append(f"{round(tint * 100):g} % black on {label} {shown}{more}") + return check + + +def read_pdf(path: Path) -> tuple[list[tuple[float, float]], list[int], list[bytes]]: + """Each page's size, number of annotations and decoded content.""" + reader = PdfReader(path) + sizes, annotations, contents = [], [], [] + for page in reader.pages: + box = page.mediabox + sizes.append((float(box.width), float(box.height))) + annots = page.get("/Annots") + annotations.append(len(annots.get_object()) if annots else 0) + content = page.get_contents() + contents.append(content.get_data() if content is not None else b"") + return sizes, annotations, contents + + +def tool_output(*command: str) -> str: + if shutil.which(command[0]) is None: + raise click.ClickException(f"{command[0]} not found - install poppler") + return subprocess.run(command, capture_output=True, text=True, check=False).stdout + + +def page_words(path: Path) -> dict[int, list[tuple]]: + """The words of each page with their boxes, from `pdftotext -bbox`.""" + words: dict[int, list[tuple]] = {} + for number, page in enumerate(tool_output("pdftotext", "-bbox", str(path), "-").split(" list[Check]: + """All checks of a PDF.""" + sizes, annotations, contents = read_pdf(path) + size_map = dict(enumerate(sizes, 1)) + tints: dict[int, set[float]] = {} + painted: dict[int, bool] = {} + for number, content in enumerate(contents, 1): + tints[number], painted[number] = area_tints(content) + images = parse_pdfimages(tool_output("pdfimages", "-list", str(path))) + words = page_words(path) + checks = [ + check_page_count(len(sizes), max_pages), + check_page_sizes(sizes), + check_fonts(parse_pdffonts(tool_output("pdffonts", str(path)))), + check_images(images, pdfx), + check_soft_masks(images), + check_transparency(read_transparency(path)), + check_annotations(annotations), + check_page_numbers(words, size_map), + check_blank_pages(words, size_map, painted), + ] + if pdfx: + # Grey areas are judged before the conversion: a neutral grey in CMYK + # mixes all four inks, which the tint estimate reads as lighter than it prints. + checks.append(check_pdfx(path, condition)) + else: + checks.append(check_light_areas(tints)) + return checks + + +def print_report(path: Path, checks: list[Check], shown: int = 10) -> bool: + """Print the checks with their findings. Returns whether no check found an error.""" + print(f"Preflight of {path}:") + for check in checks: + status = "ok" if check.passed else ("WARNING" if check.level == "warning" else "FAILED") + count = "" if check.passed else f" - {len(check.problems)}" + print(f" {status:<8} {check.name}{count}") + for problem in check.problems[:shown]: + print(f" {problem}") + if len(check.problems) > shown: + print(f" ... and {len(check.problems) - shown} more") + return all(check.passed or check.level == "warning" for check in checks) + + +@click.command(name="pdf-preflight") +@click.argument("pdf", type=click.Path(exists=True, dir_okay=False, path_type=Path)) +@click.option( + "--pdfx", + is_flag=True, + help="Also require PDF/X-4: the marker, a FOGRA39 output intent, CMYK images. Grey areas are not judged in CMYK.", +) +@click.option("--max-pages", type=int, default=MAX_PAGES, show_default=True, help="How many pages may the book have?") +@click.option( + "--intent", + default=FOGRA39, + show_default=True, + help="Which output condition must the PDF/X-4 intent name? The greyscale preview names its grey profile.", +) +def pdf_preflight(pdf: Path, pdfx: bool, max_pages: int, intent: str) -> None: + """Check the book block of the print edition before it goes to print.""" + if not print_report(pdf, run_preflight(pdf, pdfx, max_pages, intent)): + raise click.ClickException(f"preflight found errors in {pdf}") From 9c7bb4c24111ed5f4af1bfdf5bbf90d768c2d458 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Wed, 16 Sep 2026 07:45:23 +0200 Subject: [PATCH 07/15] extend ignore list --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index e0d7d8f02..42afcc295 100644 --- a/.gitignore +++ b/.gitignore @@ -10,3 +10,5 @@ tmp .python-version dist/ .env +*.sw[a-p] +*~ From 7248f943867e85a6fe3cb74ac80a2f4311a0f76b Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Wed, 16 Sep 2026 12:59:43 +0200 Subject: [PATCH 08/15] size screenshots by the density they print at The widths came from the density a screenshot was captured at, which left 42 of them printing between 85 and 149 ppi: an image fills the text column unless a width says otherwise, however few pixels it carries. - narrow every raster image below 150 ppi: width = pixels / (column in inches * 150), floored to a whole percent, one step further down where the rounding of the density falls short - read an existing width whether it is quoted or not - write 47 widths into 21 pages; nothing prints below 150 ppi any more, and the book loses eight pages - report the width a page declares next to the density in low-resolution.tsv, so an entry says whether the image is stretched or simply too small - close P15: no screenshot had to be replaced --- .claude/docs-guidelines/repo-conventions.md | 9 +- .../extracting-data-from-a-web-api/index.md | 2 +- .../index.md | 2 +- .../index.md | 4 +- docs/build/rule-blocks/index.md | 6 +- docs/build/rule-operators/index.md | 2 +- docs/build/snowflake-tutorial/index.md | 6 +- .../define-the-interfaces/index.md | 2 +- .../link-IDS-event-to-KG-via-cmem/index.md | 8 +- docs/build/variables/index.md | 6 +- docs/build/workflows/index.md | 4 +- .../consuming-graphs-in-power-bi/index.md | 2 +- .../index.md | 2 +- docs/consume/populate-data-to-neo4j/index.md | 16 +-- .../change-passwords-and-keys/index.md | 2 +- docs/develop/packages/installation/index.md | 2 +- .../python-plugins/development/index.md | 4 +- docs/distribution/marketplace/index.md | 8 +- .../visually-authoring-ontologies/index.md | 10 +- .../workflow-trigger/index.md | 2 +- .../statement-annotations/index.md | 2 +- .../workspace-configuration/index.md | 2 +- tasks/backlog.md | 60 +++++---- tasks/spec.md | 38 +++--- tests/test_build_pdf_resolution.py | 20 +-- tests/test_image_widths.py | 91 +++++++------ tools/README.md | 2 +- tools/build_pdf.py | 30 +++-- tools/image_widths.py | 126 ++++++++++-------- 29 files changed, 264 insertions(+), 206 deletions(-) diff --git a/.claude/docs-guidelines/repo-conventions.md b/.claude/docs-guidelines/repo-conventions.md index bbda5f6c3..ac38f16de 100644 --- a/.claude/docs-guidelines/repo-conventions.md +++ b/.claude/docs-guidelines/repo-conventions.md @@ -127,10 +127,11 @@ Do not write `index.html` stubs under `docs/`. - `class="bordered"` on every product screenshot (436 uses in the tree). - `width="50%"` or `width="70%"` for dialogs and modals, no width for full-screen views. Keep the width consistent within one page. -- A screenshot narrower than the page needs a width, or the print edition prints it too small or too - coarse. `poetry run dec-tool image-widths` lists raster images without one in pages that are not - generated, and `--fix` writes `width="NN%"` from the pixel width and the capture density (a Retina - capture at 144 dpi counts half), against the 16 cm print column. An image as wide as the page gets none. +- A screenshot needs a width small enough for its pixels, or it prints coarse: an image of 900 pixels + across the full 16 cm column prints at 143 ppi. `poetry run dec-tool image-widths` lists every raster + image in a page that is not generated that prints below 150 ppi, and `--fix` writes the `width="NN%"` + that reaches it — the pixel width divided by the column and the target, rounded down. The same width + then governs the site. - `.off-glb` opts an image out of the glightbox lightbox — used for inline icons and decorative images. - File names: lowercase, hyphen-separated, descriptive (style guide 5.5). - Delete screenshots that are no longer referenced. Orphan check for one page directory: diff --git a/docs/build/extracting-data-from-a-web-api/index.md b/docs/build/extracting-data-from-a-web-api/index.md index 975525b72..54e7005af 100644 --- a/docs/build/extracting-data-from-a-web-api/index.md +++ b/docs/build/extracting-data-from-a-web-api/index.md @@ -95,7 +95,7 @@ To create a JSON-to-RDF-mapping within Corporate Memory, we have to first regist 1. Click the **Create** button (top right) in the data integration workspace and select the type **JSON**. - ![Create JSON dataset](create-new-json-dataset.png) + ![Create JSON dataset](create-new-json-dataset.png){ width="71%" } 2. Upload the JSON file [repos.json](repos.json) (API response) as a Dataset into Corporate Memory. diff --git a/docs/build/lift-data-from-json-and-xml-sources/index.md b/docs/build/lift-data-from-json-and-xml-sources/index.md index c3e97ec47..3098a6f75 100644 --- a/docs/build/lift-data-from-json-and-xml-sources/index.md +++ b/docs/build/lift-data-from-json-and-xml-sources/index.md @@ -309,7 +309,7 @@ The transformation defines how an input dataset (e.g.: JSON or XML) will be tran === "JSON" - ![Examples of target data JSON](ldfjaxs-json-examples-target-data.png){ class="bordered" } + ![Examples of target data JSON](ldfjaxs-json-examples-target-data.png){ class="bordered" width="64%" } === "XML" diff --git a/docs/build/loading-jdbc-datasets-incrementally/index.md b/docs/build/loading-jdbc-datasets-incrementally/index.md index 142a0c893..bd38907f6 100644 --- a/docs/build/loading-jdbc-datasets-incrementally/index.md +++ b/docs/build/loading-jdbc-datasets-incrementally/index.md @@ -37,13 +37,13 @@ To extract data from a relational database, you need to first register a **JDBC 1. In Corporate Memory, click Projects under **BUILD** in the navigation on the left side of the page. - ![Menu BUILD > Projects](incr-jdbc-menu-build-projects.png){ class="bordered" width="45%" } + ![Menu BUILD > Projects](incr-jdbc-menu-build-projects.png){ class="bordered" width="29%" } 2. Click **Create** at the top of the page. 3. In the Create new item window, select Project and click Add. The Create new item of type Project window appears. 4. In the Create new item window, select Dataset under **Item Type**, search for JDBC endpoint, and click **Add**. - ![Create JDBC Dataset](create-dataset-JDBC.png){ class="bordered" } + ![Create JDBC Dataset](create-dataset-JDBC.png){ class="bordered" width="71%" } 5. Provide the required configuration details for the JDBC endpoint: - **Label**: Provide a table name. diff --git a/docs/build/rule-blocks/index.md b/docs/build/rule-blocks/index.md index 8efb54119..a000e09dd 100644 --- a/docs/build/rule-blocks/index.md +++ b/docs/build/rule-blocks/index.md @@ -89,7 +89,7 @@ Dragging **New input port** onto the canvas creates a port as well. Existing ports are listed below that entry. Click the **:material-pencil-outline: edit** icon of a port to open the **Edit input port** dialog, or the **:material-delete-outline: delete** icon to remove it: -![The Edit input port dialog](rule-blocks-input-port.png "The Edit input port dialog"){ width="95%" } +![The Edit input port dialog](rule-blocks-input-port.png "The Edit input port dialog"){ width="62%" } **Label** : The name of the port. @@ -145,7 +145,7 @@ It has one input handle per input port, in the port order defined in the rule bl Its context menu offers the actions specific to the reference: -![The context menu of a rule block node](rule-blocks-node-menu.png "The context menu of a rule block node"){ width="50%" } +![The context menu of a rule block node](rule-blocks-node-menu.png "The context menu of a rule block node"){ width="32%" } **Clone** : Adds a second reference to the same rule block. @@ -187,7 +187,7 @@ That also means a change can break those rules, so the editor restricts what may Click the **:material-information-outline: Usage status** button next to **Save** to see the current state: -![The usage status of a rule block](rule-blocks-usage-status.png "The usage status of a rule block") +![The usage status of a rule block](rule-blocks-usage-status.png "The usage status of a rule block"){ width="75%" } While a rule block is in use: diff --git a/docs/build/rule-operators/index.md b/docs/build/rule-operators/index.md index 815172779..f39516b03 100644 --- a/docs/build/rule-operators/index.md +++ b/docs/build/rule-operators/index.md @@ -54,7 +54,7 @@ The following parameters can be set for each comparison: !!! info inline end "" - ![Confidence graph](confidence-graph.png){ width="60%" } + ![Confidence graph](confidence-graph.png){ width="38%" } The threshold is used to convert the computed distance to a confidence between -1.0 and 1.0. Links will be generated for confidences above 0 while higher confidence values imply a higher similarity between the compared entities. diff --git a/docs/build/snowflake-tutorial/index.md b/docs/build/snowflake-tutorial/index.md index 43d2d4c84..67b1084b3 100644 --- a/docs/build/snowflake-tutorial/index.md +++ b/docs/build/snowflake-tutorial/index.md @@ -1210,7 +1210,7 @@ Here you can populate some test data with the following SQL query: - Select the **target property** according to transformation requirements, for example name, id, etc., then select the **value path** according to the target property as the product name, product id etc. This step will help in mapping the data from the source to the target property. - ![image](snowflake-target-property.png){ class="bordered" } + ![image](snowflake-target-property.png){ class="bordered" width="91%" } - Type the label name **product name**, then click on **Save**. @@ -1232,7 +1232,7 @@ Here you can populate some test data with the following SQL query: Suggestion appears as below can select as per the requirement. - ![image](snowflake-suggestion-result.png){ class="bordered" } + ![image](snowflake-suggestion-result.png){ class="bordered" width="91%" } !!! note @@ -1240,7 +1240,7 @@ Here you can populate some test data with the following SQL query: - **Tick** the box to select the suggestions to be added, then click on **Add**. -![image](snowflake-tick.png){ class="bordered" } +![image](snowflake-tick.png){ class="bordered" width="93%" } ## 4. Create a knowledge graph diff --git a/docs/build/tutorial-how-to-link-ids-to-osint/define-the-interfaces/index.md b/docs/build/tutorial-how-to-link-ids-to-osint/define-the-interfaces/index.md index 61e332c7f..c936fd293 100644 --- a/docs/build/tutorial-how-to-link-ids-to-osint/define-the-interfaces/index.md +++ b/docs/build/tutorial-how-to-link-ids-to-osint/define-the-interfaces/index.md @@ -24,7 +24,7 @@ The classic Splunk interface is a set of panels, like "static table" panel. This The first dashboard to do for our use cases is the list of IoCs with classic SPL queries of analysts via a static table and allow the analyst to select one IoC via a click in the table. The dashboard with this selected row can save the ID of IoC in a global variable for the other panels in the same dashboard ([a Splunk token](https://docs.splunk.com/Documentation/Splunk/9.0.5/Viz/tokens)). When this variable (Splunk token) is changed by the user, Splunk is able to recalculate automatically the queries with this variable in the other static tables. So with this mechanism, we can print the details in the knowledge graph (with SPARQL queries) and the IoC statistics in the Splunk indexes (with SPL queries) around of one selected IoC. With these knowledge about Splunk dashboard, we proposed to analysts a first naive interface in the figure 3.
-![Figure 3. Imagine the expected Splunk dashboard with its interaction](interface.png){ width="90%" } +![Figure 3. Imagine the expected Splunk dashboard with its interaction](interface.png){ width="57%" }
Figure 3. Imagine the expected Splunk dashboard with its interaction
diff --git a/docs/build/tutorial-how-to-link-ids-to-osint/link-IDS-event-to-KG-via-cmem/index.md b/docs/build/tutorial-how-to-link-ids-to-osint/link-IDS-event-to-KG-via-cmem/index.md index cc9e47626..2e49803f6 100644 --- a/docs/build/tutorial-how-to-link-ids-to-osint/link-IDS-event-to-KG-via-cmem/index.md +++ b/docs/build/tutorial-how-to-link-ids-to-osint/link-IDS-event-to-KG-via-cmem/index.md @@ -56,17 +56,17 @@ For each investigation, an analyst selects and navigates in the data with two da
-![Figure 2: High-level investigation dashboard with the list of computers implicated in the incident](poc-app_high_level_computers.png) +![Figure 2: High-level investigation dashboard with the list of computers implicated in the incident](poc-app_high_level_computers.png){ width="68%" }
Figure 2: High-level investigation dashboard with the list of computers implicated in the incident
-![Figure 3: High-level investigation dashboard with the panel to select a specific period during an incident according to IoCs details](poc-app_high_level_period.png) +![Figure 3: High-level investigation dashboard with the panel to select a specific period during an incident according to IoCs details](poc-app_high_level_period.png){ width="69%" }
Figure 3: High-level investigation dashboard with the panel to select a specific period during an incident according to IoCs details
-![Figure 4: low-level investigation dashboard contains the command lines of Windows processus rised Suricata alerts during the period selected by the analyst in a high-level investigation dashboard](poc-app_low_level.png) +![Figure 4: low-level investigation dashboard contains the command lines of Windows processus rised Suricata alerts during the period selected by the analyst in a high-level investigation dashboard](poc-app_low_level.png){ width="73%" }
Figure 4: low-level investigation dashboard contains the command lines of Windows processus rised Suricata alerts during the period selected by the analyst in a high-level investigation dashboard
@@ -75,7 +75,7 @@ The idea is the analyst can do an investigation high-level without using a lot o To follow the calculation of investigations and free memory when one investigation is closed, we developed another dashboard "investigation_list.xml" (see figure 5). This dashboard prints the status of investigations actually in the knowledge graph. Here, each investigation is saved in tempory graphs and the analyst can create and delete them directly in Splunk.
-![Figure 5: The dashboard "investigation list" shows all the tempory graph actually in the knowledge graph. The analyst can open an investigation, see the SPL query generated when he has created an investigation and delete it when he want.](poc-app_list_investigations.png) +![Figure 5: The dashboard "investigation list" shows all the tempory graph actually in the knowledge graph. The analyst can open an investigation, see the SPL query generated when he has created an investigation and delete it when he want.](poc-app_list_investigations.png){ width="82%" }
Figure 5: The dashboard "investigation list" shows all the tempory graph actually in the knowledge graph. The analyst can open an investigation, see the SPL query generated when he has created an investigation and delete it when he want.
diff --git a/docs/build/variables/index.md b/docs/build/variables/index.md index 1257862b4..213c19e85 100644 --- a/docs/build/variables/index.md +++ b/docs/build/variables/index.md @@ -114,7 +114,7 @@ In order to add project variables, login to eccenca Corporate Memory, select the On the right side of the page you can see the variable section Click on :eccenca-item-add-artefact: to add the variables. -![](di-var-add-variable.png){ class="bordered" } +![](di-var-add-variable.png){ class="bordered" width="80%" } The variable definition dialog opens. Type the name, value and description of the variable. @@ -138,7 +138,7 @@ Type name as `email_ids`, in values we have updated all the email id’s of the The email_ids variable is added as shown below. - ![](di-var-email-defined.png){ class="bordered" } + ![](di-var-email-defined.png){ class="bordered" width="84%" } ## Execution Variables @@ -225,7 +225,7 @@ Select the `project.email_ids` and include the double curly brackets at last. Same as above, add the variable click on :eccenca-item-add-artefact: and type name as `password`, value as `xmhnjnnjkmnlbbhbvfhnbjkm` and in the description `Gmail account password` as shown below then click on **Add**. -![](di-var-password-definition.png){ class="bordered" width="95%" } +![](di-var-password-definition.png){ class="bordered" width="76%" } !!! success "Step Result" diff --git a/docs/build/workflows/index.md b/docs/build/workflows/index.md index 30fc40b19..313f42bf2 100644 --- a/docs/build/workflows/index.md +++ b/docs/build/workflows/index.md @@ -55,7 +55,7 @@ The **Clear Dataset** operator empties the dataset connected to its output befor Place the Clear Dataset operator in the workflow and connect its output to the dataset that should be cleared. The operator takes no data inputs; connect it using a **dependency connection** from the upstream node that must complete first, or leave it unconnected if the dataset should be cleared before any subsequent nodes execute. -![Clear Datasets](wf-clear-datasets.png){ width="90%" } +![Clear Datasets](wf-clear-datasets.png){ width="57%" } Some datasets historically provided their own clear attributes (e.g., `Clear graph before workflow execution` on the Knowledge Graph dataset). These per-dataset attributes are **deprecated** and should no longer be used. Use the Clear Dataset operator instead, which works uniformly across all dataset types. @@ -67,7 +67,7 @@ The operators of a workflow can be reconfigured completely in the context of a w During its execution, new parameters are loaded from any possible source and translated by a transformation task to allow an injection into the dataset configuration that overwrites originally set parameters. To reconfigure a workflow operator, the transformation task has to be connected to the red dot at the top of this operator as shown in the following image: -![Workflow config port](wf-config-port.png) +![Workflow config port](wf-config-port.png){ width="86%" } Although this feature has been developed to support the ingestion of database deltas, the possible applications are various since any parameter can be overwritten to make workflow operators even more dynamic and reusable in various contexts. The incremental ingestion of database content that was implemented as a first use-case can be found the application section of this page. diff --git a/docs/consume/consuming-graphs-in-power-bi/index.md b/docs/consume/consuming-graphs-in-power-bi/index.md index 68f2d7a8b..25050c158 100644 --- a/docs/consume/consuming-graphs-in-power-bi/index.md +++ b/docs/consume/consuming-graphs-in-power-bi/index.md @@ -37,7 +37,7 @@ The latest (unsigned) version of our Power-BI-Connector is available from its so - The registry path is `HKEY_LOCAL_MACHINE\Software\Policies\Microsoft\Power BI Desktop` . Make sure the path exists, or create it. - Add a new value under the path specified above. The type should be “Multi-String Value” ( `REG_MULTI_SZ` ), and it should be called `TrustedCertificateThumbprints` - Add the thumbprints of the certificates you want to trust. You can add multiple certificates by using “\0” as a delimiter, or in the registry editor, right click → modify and put each thumbprint on a new line. - ![PowerBI-Registry-Editor.png](22-1-PowerBI-Registry-Editor.png) + ![PowerBI-Registry-Editor.png](22-1-PowerBI-Registry-Editor.png){ width="95%" } - (Re-)Start Power BI Desktop If you wish to automate this setup you can use the reg windows command line tool to make this entry like: diff --git a/docs/consume/consuming-graphs-with-sql-databases/index.md b/docs/consume/consuming-graphs-with-sql-databases/index.md index 096c825a8..8d3e5bb50 100644 --- a/docs/consume/consuming-graphs-with-sql-databases/index.md +++ b/docs/consume/consuming-graphs-with-sql-databases/index.md @@ -29,7 +29,7 @@ In the following, we have a more detailed look at each of the three operators. Create a dataset of the type Knowledge Graph (embedded) and set the graph parameter to the URI of the graph that contains the resources to be exported: -![create-dataset](22-1-1-create-dataset.png) +![create-dataset](22-1-1-create-dataset.png){ width="70%" } ### Create Transformation diff --git a/docs/consume/populate-data-to-neo4j/index.md b/docs/consume/populate-data-to-neo4j/index.md index d29c7ae79..656163eaf 100644 --- a/docs/consume/populate-data-to-neo4j/index.md +++ b/docs/consume/populate-data-to-neo4j/index.md @@ -36,20 +36,20 @@ This tutorial uses the _Multi CSV Zip_ format to efficiently handle the followin To upload multiple files together as an input: 1. In Corporate Memory, click Projects under **BUILD** in the navigation on the left side of the page. - ![Menu > BUILD > Projects](pdtn-menu-projects.png){width="50%"} + ![Menu > BUILD > Projects](pdtn-menu-projects.png){width="29%"} 2. Click Create at the top of the page. 3. In Create new item window, select Project and click Add. The Create new item of type Project window appears. 4. Fill in the required details such as Title and Description.  Alternatively, import the existing project by clicking Import Project File and selecting the file from your system. 5. Click Create. Your project (Northwind) is created. - ![Project Northwind](pdtn-project-northwind.png) + ![Project Northwind](pdtn-project-northwind.png){ width="79%" } 6. In your project, click Create Item. 7. In the Create new item window, select Multi CSV ZIP and click Add. - ![Create Multi CSV Zip dataset](pdtn-create-dataset-multi-csv.png) + ![Create Multi CSV Zip dataset](pdtn-create-dataset-multi-csv.png){ width="64%" } 8. Specify a Label of the dataset in the Create new item of type Multi CSV ZIP window. 9. Select the Upload new file option as you have the files. The Multi CSV ZIP file containing the above-listed files is available here. If it is an existing project, you can select the files from the project. For the remaining parameters, the default settings are used. 10. Click Create. You can see the message northwind.zip was successfully uploaded in Green. 11. You can see the Multi CSV ZIP file is uploaded with the datasets, and the item has been created. - ![Upload file to dataset](pdtn-create-dataset-upload.png) + ![Upload file to dataset](pdtn-create-dataset-upload.png){ width="71%" } 12. Click the Play button and review the dataset in the Data preview section. You can see the contents of the loaded zip file consisting of the CSV files introduced above. ## 2 Create a Neo4j dataset @@ -59,7 +59,7 @@ Perform the following steps to create a Neo4j dataset: 1. In your existing project, click Create to create a new item. 2. In the item category Dataset select Neo4j. - ![Create Neo4j dataset](pdtn-create-dataset-neo4j.png) + ![Create Neo4j dataset](pdtn-create-dataset-neo4j.png){ width="68%" } 3. Click Add. 4. Enter the following details: - Label: _Name of the item_.   @@ -81,7 +81,7 @@ The provided vocabulary is inspired by this structure from the original Neo4j tu The vocabulary for the Northwind project can be visualized like this: -![Northwind vocabulary](pdtn-vocabulary.png) +![Northwind vocabulary](pdtn-vocabulary.png){ width="96%" } ## 4 Create Transformations @@ -263,7 +263,7 @@ Click the Play icon to validate the results. The nodes and edges created are sto ## 6 Results in Neo4j -![Result in Neo4j](pdtn-result-in-neo4j.png) +![Result in Neo4j](pdtn-result-in-neo4j.png){ width="89%" } ## 7 Results in Knowledge Graph @@ -272,4 +272,4 @@ Optionally, you can use the same transformation and workflow to render the resul 1. Add a Knowledge Graph dataset and use this as an additional target in your workflow: ![Add Knowledge Graph output](pdtn-add-kg-dataset-into-workflow.png) 2. The results can then be reviewed in the Knowledge Graph module, e.g., explored visually: - ![Results in Corporate Memory](pdtn-ontodia-result-view.png) + ![Results in Corporate Memory](pdtn-ontodia-result-view.png){ width="72%" } diff --git a/docs/deploy-and-configure/configuration/keycloak/change-passwords-and-keys/index.md b/docs/deploy-and-configure/configuration/keycloak/change-passwords-and-keys/index.md index 8eed7ed7a..498504306 100644 --- a/docs/deploy-and-configure/configuration/keycloak/change-passwords-and-keys/index.md +++ b/docs/deploy-and-configure/configuration/keycloak/change-passwords-and-keys/index.md @@ -50,7 +50,7 @@ To change the default passwords, select the Cmem Realm and open Users in the lef ![users-lookup](22-1-users-lookup.png) -![user-details](22-1-user-details.png) +![user-details](22-1-user-details.png){ width="75%" } Then, select "View all users" and choose an account you want to change the password for (we start with admin) diff --git a/docs/develop/packages/installation/index.md b/docs/develop/packages/installation/index.md index 49788a23a..0da014ce3 100644 --- a/docs/develop/packages/installation/index.md +++ b/docs/develop/packages/installation/index.md @@ -74,7 +74,7 @@ Depending on the content types inside it, an installed package appears in differ !!! info inline "" - ![Example: Graphs](example-vocabulary.png "Example: Graphs") + ![Example: Graphs](example-vocabulary.png "Example: Graphs"){ width="68%" } **Graphs** such as data graphs but also **Vocabularies** or **Shapes Catalogs** are listed in [**Explore > Graphs**](../../../explore-and-author/graph-exploration/index.md#graphs). diff --git a/docs/develop/python-plugins/development/index.md b/docs/develop/python-plugins/development/index.md index 0c9c13ad7..a1b95218b 100644 --- a/docs/develop/python-plugins/development/index.md +++ b/docs/develop/python-plugins/development/index.md @@ -350,7 +350,7 @@ The method returns a list of `Autocompletion` objects, which represent the possi The [cmem-plugin-base](https://github.com/eccenca/cmem-plugin-base/) package describes [context objects](https://github.com/eccenca/cmem-plugin-base/blob/main/cmem_plugin_base/dataintegration/context.py), which are passed to the plugin depending on the executed method. -![context-api-flow-diagram](23-1-context-api-flow-diagram.png) +![context-api-flow-diagram](23-1-context-api-flow-diagram.png){ width="75%" } ### Basic Understanding @@ -453,7 +453,7 @@ The following image shows these terms and their relationships. (1) 1. The concrete implementation details of entities can be found in the [entity module](https://github.com/eccenca/cmem-plugin-base/blob/main/cmem_plugin_base/dataintegration/entity.py) of the cmem-plugin-base package. -![entities-flow-diagram](22-2-entities-flow-diagram.png) +![entities-flow-diagram](22-2-entities-flow-diagram.png){ width="90%" } | Class | Description | | -------------- | ------------------------------------------------------------------ | diff --git a/docs/distribution/marketplace/index.md b/docs/distribution/marketplace/index.md index 3865bc5b3..dd7b27a27 100644 --- a/docs/distribution/marketplace/index.md +++ b/docs/distribution/marketplace/index.md @@ -61,7 +61,7 @@ They can be combined. !!! info inline "" - ![The open Package type drop-down](marketplace-filter-package-type.png "The open Package type drop-down"){ width="35%" } + ![The open Package type drop-down](marketplace-filter-package-type.png "The open Package type drop-down"){ width="21%" } #### Package type @@ -73,7 +73,7 @@ Show only **Vocabulary** or only **Project** packages, see [Package Types](#pack !!! info inline "" - ![The open License drop-down](marketplace-filter-license.png "The open License drop-down"){ width="50%" } + ![The open License drop-down](marketplace-filter-license.png "The open License drop-down"){ width="32%" } #### License @@ -93,7 +93,7 @@ Show only packages whose name or description contains the entered keyword. !!! info inline "" - ![The Installed toggle, switched on](marketplace-filter-installed.png "The Installed toggle, switched on"){ width="20%" } + ![The Installed toggle, switched on](marketplace-filter-installed.png "The Installed toggle, switched on"){ width="14%" } #### Installed @@ -105,7 +105,7 @@ Switch on to show only the packages currently installed in your instance. !!! info inline "" - ![The open Marketplace drop-down](marketplace-filter-server.png "The open Marketplace drop-down"){ width="60%" } + ![The open Marketplace drop-down](marketplace-filter-server.png "The open Marketplace drop-down"){ width="38%" } #### Marketplace (top right) diff --git a/docs/explore-and-author/bke-module/visually-authoring-ontologies/index.md b/docs/explore-and-author/bke-module/visually-authoring-ontologies/index.md index fcc002f0e..d203c85d2 100644 --- a/docs/explore-and-author/bke-module/visually-authoring-ontologies/index.md +++ b/docs/explore-and-author/bke-module/visually-authoring-ontologies/index.md @@ -46,7 +46,7 @@ To create a new ontology graph: 1. In Corporate Memory, click **Knowledge graphs** under **EXPLORE** in the navigation on the left side of the page. - ![](explore-kgs.png){ class="bordered" width="50%" } + ![](explore-kgs.png){ class="bordered" width="29%" } 2. In the **Graphs** drop-down menu, click the **(+)** button and select **New Ontology (owl:Ontology)**. @@ -62,7 +62,7 @@ To create a new ontology graph: 1. In Corporate Memory, click **Business Knowledge Editor** under **EXPLORE** in the navigation on the left side of the page. - ![](explore-bke.png){ class="bordered" width="50%" } + ![](explore-bke.png){ class="bordered" width="29%" } 2. Select the target graph using the drop-down menu. @@ -82,7 +82,7 @@ New elements can be created from the entries listed in **Classes** on the left s 1. Drag and drop **Class** from the bottom left list into the canvas. - ![](classes.png){ class="bordered" width="50%" } + ![](classes.png){ class="bordered" width="28%" } !!! info @@ -130,7 +130,7 @@ New elements can be created from the entries listed in **Classes** on the left s 7. Drag and drop **New Class** into the canvas. - ![](drag-drop-new-class.png){ class="bordered" width="50%" } + ![](drag-drop-new-class.png){ class="bordered" width="49%" } 8. Click the newly created class to open its form, then fill out the required fields and any optional fields you want to define. @@ -181,7 +181,7 @@ Save the changes as a named visualization so you can edit your classes and prope 3. Enter a name for the visualization. - ![](save-changes.png) + ![](save-changes.png){ width="76%" } 4. Click **Save**. diff --git a/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/workflow-trigger/index.md b/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/workflow-trigger/index.md index 6099518f7..ac311bfb0 100644 --- a/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/workflow-trigger/index.md +++ b/docs/explore-and-author/graph-exploration/building-a-customized-user-interface/workflow-trigger/index.md @@ -36,7 +36,7 @@ A `shui:WidgetIntegration` resource needs to be created from the previously defi Once the widget is created, it can be attached to a Node Shape by using a special-purpose non-validating [Widget property](../node-shapes/index.md#widgets). -![](adding_widget_to_node.png){ width="95%" } +![](adding_widget_to_node.png){ width="62%" } ## Payload Structure diff --git a/docs/explore-and-author/graph-exploration/statement-annotations/index.md b/docs/explore-and-author/graph-exploration/statement-annotations/index.md index cfb39d898..94cbd6779 100644 --- a/docs/explore-and-author/graph-exploration/statement-annotations/index.md +++ b/docs/explore-and-author/graph-exploration/statement-annotations/index.md @@ -76,7 +76,7 @@ These Annotation Resources are based on specific Shapes which are enabled as Sta Reification Resources as well as Annotation Resources are managed in a Statement Annotation Graph, which need to be configured on a Graph as well as imported to this Graph. The following illustration depicts this schema with boxes and arrows: -![](20-10-StatementAnnotationSchema.png){ class="bordered" } +![](20-10-StatementAnnotationSchema.png){ class="bordered" width="91%" } !!! note "Some notes on this:" diff --git a/docs/explore-and-author/workspace-configuration/index.md b/docs/explore-and-author/workspace-configuration/index.md index dd31c57a6..14e03ee4d 100644 --- a/docs/explore-and-author/workspace-configuration/index.md +++ b/docs/explore-and-author/workspace-configuration/index.md @@ -25,7 +25,7 @@ Click the drop-down list and click the Application view you want to open. Click the :eccenca-module-workspace-configuration: **Application view configuration** icon in the main menu. -![Configuration module](configuration.png){ class="bordered" width="45%" } +![Configuration module](configuration.png){ class="bordered" width="29%" } Select the Application view from the dropdown menu to configure. diff --git a/tasks/backlog.md b/tasks/backlog.md index f4d3fbced..69fa3cb1c 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -353,7 +353,7 @@ pages blank. Non-zero exit on a violation; run by `task pdf:print`, on the norma --- -## P15 - Low-resolution originals +## P15 - Low-resolution originals - **done** 49 images were below 150 ppi at printed size; resampling (P12) hides that from the preflight but adds no detail. After P18, 43 images in the printed pages that are not generated print below 150 ppi, 37 of them @@ -363,12 +363,18 @@ with a fresh screenshot, or accept it in spec §8. **Verify:** every entry replaced or accepted. **Est:** medium, mostly content. **Depends on:** P12, P24. +Done 2026-09-16 by the revised P24: no screenshot had to be replaced. Narrowing the 42 entries to the +width their pixels support empties the report, so `accepted-low-resolution` in `tools/pdf/print.yml` +stays an empty list, and the report keeps watch over new screenshots. + Tooling done 2026-09-15: the print build collects every original below 150 ppi at its printed size (`resolve_images(..., low_resolution)`), leaves out those listed under `accepted-low-resolution` in -`tools/pdf/print.yml`, writes the rest lowest first to `dist/pdf/print/low-resolution.tsv` and prints -their count. Accepting moved from spec §8 to `print.yml`, next to the other print settings. **Open, content -work:** 42 images, 85-149 ppi, most in `consume/populate-data-to-neo4j` (8), `explore-and-author/bke-module` -(5) and `distribution/marketplace` (4). +`tools/pdf/print.yml`, writes the rest lowest first to `dist/pdf/print/low-resolution.tsv` - density, the +width the page declares (the `width="NN%"` of the Markdown source, empty where there is none) and the +image - and prints their count. Accepting moved from spec §8 to `print.yml`, next to the other print +settings. The 42 entries it listed - 85 to 149 ppi, most in `consume/populate-data-to-neo4j` (8), +`explore-and-author/bke-module` (5) and `distribution/marketplace` (4) - are resolved by the revised P24: +the report is empty. --- @@ -581,25 +587,31 @@ with more than 20 parameters. ## P24 - Image widths in the sources - **done** -Done 2026-09-15: `dec-tool image-widths` (`tools/image_widths.py`) checks without `--fix` and fails when -an image lacks a width. Refinement: an image at 100 % or wider gets **no** width - the column caps it in -print anyway, and 456 such images would otherwise have been pinned to the article width on the site, -upscaling the narrower ones there. `--fix` wrote 24 widths into 14 pages (20 % to 95 %), 10 of them on -the IDS/OSINT tutorial, which the print edition omits; rumdl clean. Below 150 ppi in the print edition: -43 before, 42 after - an image without a declared density now prints at 96 instead of 72 ppi, still -below 150, so P15 keeps its list. `task check` does not run the check yet. 15 unit tests. - -Spec §11, D18. `dec-tool image-widths --check|--fix` writes `{ width="NN%" }` for raster images without a -width in pages that are not generated: the pixel width divided by the capture scale and by the full page -width - the 16 cm text column, 605 CSS pixels - rounded to 5 %, at most 100 %. The capture scale is 2 for -144 dpi (macOS Retina), otherwise the declared density / 96, and 1 without one. Optionally `task check` -runs `--check`. - -**Verify:** unit tests for capture scale, percentage, rounding and the skip rules; after `--fix` no raster -image in a page that is not generated lacks a width; spot checks of the changed pages on the site and in -the print edition; the count below 150 ppi in the print edition before and after - no image prints less -sharply than before. -**Est:** medium, touches many pages. **Depends on:** nothing. P15 follows it. +Revised on 2026-09-16, once the report of P15 made the effect visible: the width follows from the density +an image *prints* at, not from the density it was *captured* at. `dec-tool image-widths` +(`tools/image_widths.py`) narrows every raster image that prints below 150 ppi in the 16 cm column, in +pages that are not generated, and leaves the rest as it is. + +- **Width:** `pixels / (6.3 inches * 150)`, rounded down to a whole percent - not to a multiple of 5. The + width a page already declares cancels out of `declared * density / target`, so the pixels alone decide; + one further step down covers the case where the rounding of the density leaves it a pixel short. +- **Measured at** the width the page declares, quoted or not - two of 187 are written `width=11%` - and an + image without one fills the column, so it counts as 100 %. +- **No floor:** the smallest results are `marketplace-filter-installed.png` at 14 % and + `marketplace-filter-package-type.png` at 21 %. Both show a single snippet of a dialog, so the user chose + the calculated value over a floor. +- **Scope:** raster images only; SVGs, remote images and fenced code stay untouched, and generated pages + belong to their generators - none of them held an image below the target. + +**Verified:** `--fix` wrote 47 widths into 21 pages; the report of P15 is empty afterwards and the book +shrinks from 872 to 864 pages. The gate was a throwaway PDF of just those images at their new widths. +Five of the 47 sit in the IDS/OSINT tutorial, which print omits, so they change the site only. rumdl +clean; the unit tests cover the density, the target width, and the rewriting of quoted, unquoted and +missing width attributes. + +**Superseded:** the first implementation (2026-09-15, D18 as written) took the natural width from the +capture scale, rounded it to 5 % and skipped anything at 100 %. Its 24 widths stay where the new rule does +not narrow them further. `task check` does not run the check yet. --- diff --git a/tasks/spec.md b/tasks/spec.md index ca4867be1..b511cf546 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -566,7 +566,7 @@ P19-P24 and P13 are implemented; P15 keeps 42 images to replace or accept. | Facing cards in a two-column grid should be equally high | each card as high as its text | one height per row | P22 | | The operator reference should print descriptions and parameters | overview tables, `list` mode (§5) | one compact entry per operator, replacing the overview tables (D17) | P23 | | PDF/X-4 and CMYK | specified, not built (§7) | unchanged; done, with transparency rendered to images first | P13 | -| Low-resolution images mostly lack `width` in the Markdown | an image without `width` prints at its declared density | widths computed from pixels and capture density against the full page width, written into the sources (D18); done, 42 originals stay below 150 ppi | P24, then P15 | +| Low-resolution images mostly lack `width` in the Markdown | an image without `width` prints at its declared density | widths written into the sources so every image prints at 150 ppi or better (D18, revised 2026-09-16); done, and P15 with it | P24, then P15 | ### Footer: the part label @@ -732,22 +732,26 @@ images, 352 of them without `width`. The declared densities: none on 231, 144 dp captures on macOS), 120 dpi on 80, 96 dpi on 8, 192 dpi on 1. 43 images print below 150 ppi, 37 of them without `width`. -Decided (D18) - `dec-tool image-widths`, with `--check` and `--fix`, writes the missing `{ width="NN%" }` -into the Markdown of pages that are not generated: - -- **Capture scale:** 2 for 144 dpi, the density of a Retina capture on macOS (2 × 72). Any other declared - density is divided by 96, so 120 dpi gives 1.25 and 192 dpi gives 2. An image that declares none, 72 or - 96 dpi has scale 1. -- **Natural width:** the pixel width divided by the capture scale, in CSS pixels at 96 per inch. -- **Width:** the natural width as a share of the full page width - the 16 cm text column, 605 CSS pixels - - rounded to 5 % and at most 100 %. -- **Scope:** raster images only. SVGs and images that already have a width stay as they are. -- **Check:** `task check` may run `--check`, so a new screenshot gets its width. - -The same percentage serves the site and the PDF. In print an image then keeps its natural size and prints -at its capture scale times 96 ppi - 96 ppi at scale 1, 192 ppi for a Retina capture - and an image wider -than the column still fills it. No image prints less sharply than today, when Typst assumes 72 dpi for an -image without a declared density. P15 handles what stays below 150 ppi. +Decided as D18 on 2026-09-15 from the capture density, and **revised on 2026-09-16** once the report of +P15 showed what each image actually prints at. The rule is the printed density: `dec-tool image-widths`, +with `--fix`, narrows every raster image that prints below 150 ppi in pages that are not generated. + +- **Width:** the pixel width divided by the 6.3 inches of the text column and by the 150 ppi target, + rounded **down to a whole percent**, with one further step down where the rounding of the density would + leave it short. The width a page declares cancels out of `declared × density ÷ target`, so the pixels + alone decide it. +- **Measured at** the width the page declares, quoted or not; an image without one fills the column and + counts as 100 %. +- **Scope:** raster images only. SVGs, remote images, fenced code and images already at 150 ppi or more + stay as they are, as do generated pages. +- **No floor:** an image whose pixels only support a thumbnail gets the thumbnail; the two smallest are + 14 % and 21 %, each a snippet of a dialog. +- **Check:** `task check` may run the check, so a new screenshot is caught. + +The same percentage serves the site and the PDF: on paper the image prints at 150 ppi or better, and the +site shows it at the share of the article it fills on the page. This closes P15 - `--fix` wrote 47 widths +into 21 pages and the report is empty - rather than replacing screenshots. The superseded rule, natural +width from the capture scale rounded to 5 %, left 42 images below the target. ### Decisions diff --git a/tests/test_build_pdf_resolution.py b/tests/test_build_pdf_resolution.py index 180e191fb..6ae3f0769 100644 --- a/tests/test_build_pdf_resolution.py +++ b/tests/test_build_pdf_resolution.py @@ -17,16 +17,19 @@ def test_originals_below_150_ppi_at_their_printed_size_are_listed_once(tmp_path, Image.new("RGB", (300, 100), "white").save(site / "build/small.png") Image.new("RGB", (2000, 100), "white").save(site / "build/wide.png") Image.new("RGB", (600, 100), "white").save(site / "build/half.png") + Image.new("RGB", (100, 50), "white").save(site / "build/narrow.png") doc = BeautifulSoup( '

' - '

', + '' + '

', "html.parser", ) - low: dict[str, int] = {} + low: dict[str, tuple[int, str]] = {} resolve_images(doc, site, Counter(), [], Path("print-images"), low_resolution=low) # 300 px without a declared density print at 300 pt, 72 ppi; 2000 px fill the - # 16 cm column at 317 ppi; 600 px at 50 % of the column print at 190 ppi. - assert low == {"build/small.png": 72} + # 16 cm column at 317 ppi; 600 px at 50 % of the column print at 190 ppi; and + # 100 px at 25 % of it print at 64 ppi, listed with the width of the source. + assert low == {"build/small.png": (72, ""), "build/narrow.png": (64, "25%")} def test_the_screen_edition_lists_nothing(tmp_path, monkeypatch): @@ -35,7 +38,7 @@ def test_the_screen_edition_lists_nothing(tmp_path, monkeypatch): (site / "build").mkdir(parents=True) Image.new("RGB", (300, 100), "white").save(site / "build/small.png") doc = BeautifulSoup('', "html.parser") - low: dict[str, int] = {} + low: dict[str, tuple[int, str]] = {} resolve_images(doc, site, Counter(), [], None, low_resolution=low) assert low == {} @@ -45,9 +48,10 @@ def test_accepted_originals_are_not_listed(tmp_path): path.write_text("accepted-low-resolution:\n - build/small.png\n") accepted = load_accepted_low_resolution(path) assert accepted == {"build/small.png"} - assert unaccepted_low_resolution({"build/small.png": 72, "build/tiny.png": 40, "build/mid.png": 120}, accepted) == [ - ("build/tiny.png", 40), - ("build/mid.png", 120), + low = {"build/small.png": (72, ""), "build/tiny.png": (40, "20%"), "build/mid.png": (120, "")} + assert unaccepted_low_resolution(low, accepted) == [ + ("build/tiny.png", 40, "20%"), + ("build/mid.png", 120, ""), ] diff --git a/tests/test_image_widths.py b/tests/test_image_widths.py index bcf60449c..6185921d2 100644 --- a/tests/test_image_widths.py +++ b/tests/test_image_widths.py @@ -1,80 +1,93 @@ -"""Test the screenshot widths written into the Markdown sources (tasks/spec.md, §11, D18)""" +"""Test the screenshot widths written into the Markdown sources (tasks/spec.md, §11, D18; backlog P24)""" import pytest from PIL import Image -from tools.image_widths import add_widths, capture_scale, width_percent +from tools.image_widths import TARGET_PPI, fit_widths, printed_density, target_width @pytest.mark.parametrize( - "dpi, scale", - [(None, 1.0), (72, 1.0), (96, 1.0), (120, 1.25), (144, 2.0), (192, 2.0)], + "pixels, percent, ppi", + [ + (945, 100, 150), # the 16 cm column is 6.3 inches + (472, 50, 150), # half the column, half the pixels + (945, 50, 300), # the same image at half the width prints twice as sharp + (300, 100, 48), + ], ) -def test_the_capture_scale_follows_the_declared_density(dpi, scale): - assert capture_scale(dpi) == scale +def test_the_density_follows_from_the_width_on_the_page(pixels, percent, ppi): + assert printed_density(pixels, percent) == ppi @pytest.mark.parametrize( - "pixels, dpi, percent", + "pixels, percent, width", [ - (605, None, 100), # the full page width at scale 1 - (300, None, 50), # 49.6 % rounds to 50 - (1210, 144, 100), # a Retina capture: 605 CSS pixels - (400, 144, 35), # 200 CSS pixels: 33 % rounds to 35 - (480, 120, 65), # 384 CSS pixels: 63.5 % rounds to 65 - (20, None, 5), # never below 5 % - (2000, None, 100), # never above 100 % + (472, 100, 50), # 75 ppi at the full column: half of it reaches the target exactly + (472, 50, 50), # the declared width cancels out - the pixels decide + (904, 100, 95), # 144 ppi: floor gives 96 %, which still measures 149 + (282, 50, 29), # 90 ppi at half the column + (150, 20, 15), # a thumbnail keeps shrinking ], ) -def test_the_width_is_the_natural_width_as_a_share_of_the_full_page(pixels, dpi, percent): - assert width_percent(pixels, dpi) == percent +def test_the_width_brings_the_image_to_the_target_density(pixels, percent, width): + assert target_width(pixels, percent) == width + assert printed_density(pixels, width) >= TARGET_PPI + + +def test_an_image_too_small_for_the_target_keeps_the_smallest_width(): + assert target_width(9, 100) == 1 + assert printed_density(9, 1) < TARGET_PPI -def picture(path, width, dpi=None): - image = Image.new("RGB", (width, 10), "white") - if dpi: - image.save(path, dpi=(dpi, dpi)) - else: - image.save(path) +def picture(path, width): + Image.new("RGB", (width, 10), "white").save(path) -def test_images_without_a_width_get_one_and_the_rest_stay_as_they_are(tmp_path): - picture(tmp_path / "plain.png", 300) - picture(tmp_path / "framed.png", 600, dpi=144) - picture(tmp_path / "sized.png", 300) - picture(tmp_path / "titled.png", 400) - picture(tmp_path / "wide.png", 1500) +def test_images_that_print_too_coarse_are_narrowed_and_the_rest_stay_as_they_are(tmp_path): + picture(tmp_path / "plain.png", 472) + picture(tmp_path / "framed.png", 472) + picture(tmp_path / "quoted.png", 282) + picture(tmp_path / "unquoted.png", 282) + picture(tmp_path / "sharp.png", 1200) + picture(tmp_path / "titled.png", 472) picture(tmp_path / "coded.png", 300) (tmp_path / "diagram.svg").write_text("") text = ( "# Page\n\n" "![Plain](plain.png)\n\n" '![Framed](framed.png){ class="bordered" }\n\n' - '![Sized](sized.png){ class="bordered" width="70%" }\n\n' + '![Quoted](quoted.png){ class="bordered" width="50%" }\n\n' + "![Unquoted](unquoted.png){ width=50% }\n\n" + "![Sharp](sharp.png)\n\n" '![Titled](titled.png "A title")\n\n' - "![Wide](wide.png)\n\n" "![Remote](https://example.org/remote.png)\n\n" "![Vector](diagram.svg)\n\n" "![Missing](gone.png)\n\n" "```markdown\n![Coded](coded.png)\n```\n" ) - result, changes = add_widths(text, tmp_path) - # A full-width image gets no width: it fills the column in print either way. + result, changes = fit_widths(text, tmp_path) assert result == ( "# Page\n\n" '![Plain](plain.png){ width="50%" }\n\n' '![Framed](framed.png){ class="bordered" width="50%" }\n\n' - '![Sized](sized.png){ class="bordered" width="70%" }\n\n' - '![Titled](titled.png "A title"){ width="65%" }\n\n' - "![Wide](wide.png)\n\n" + '![Quoted](quoted.png){ class="bordered" width="29%" }\n\n' + '![Unquoted](unquoted.png){ width="29%" }\n\n' + "![Sharp](sharp.png)\n\n" + '![Titled](titled.png "A title"){ width="50%" }\n\n' "![Remote](https://example.org/remote.png)\n\n" "![Vector](diagram.svg)\n\n" "![Missing](gone.png)\n\n" "```markdown\n![Coded](coded.png)\n```\n" ) - assert changes == [("plain.png", 50), ("framed.png", 50), ("titled.png", 65)] + assert [(source, percent, width) for source, percent, width, _, _ in changes] == [ + ("plain.png", 100, 50), + ("framed.png", 100, 50), + ("quoted.png", 50, 29), + ("unquoted.png", 50, 29), + ("titled.png", 100, 50), + ] -def test_a_page_whose_images_all_have_widths_is_unchanged(tmp_path): - picture(tmp_path / "sized.png", 300) +def test_a_page_whose_images_print_sharply_is_unchanged(tmp_path): + picture(tmp_path / "sized.png", 472) text = '![Sized](sized.png){ width="40%" }\n' - assert add_widths(text, tmp_path) == (text, []) + assert fit_widths(text, tmp_path) == (text, []) diff --git a/tools/README.md b/tools/README.md index e77ed611d..453649137 100644 --- a/tools/README.md +++ b/tools/README.md @@ -15,7 +15,7 @@ supported entry points and pass the right options. | `pdf-normalize` | `task pdf:print -- --normalize`, `task pdf:print -- --gray` | Write the print PDF as PDF/X-4 in CMYK with Ghostscript, checking the result for the PDF/X marker; the ISO Coated v2 profile is fetched into `dist/icc/`. `--gray` writes a greyscale preview of the black-and-white print instead, PDF/X-4 by the grey profile Ghostscript ships | | `pdf-preflight` | `task pdf:print` | Check the book block for print: A4, even page count, embedded fonts, image resolution, soft masks, annotations, page-number position, blank pages, light grey areas | | `check-zensical-output` | `task check:output` | Inspect the built `site/` and fail if a feature we reimplemented for Zensical regressed | -| `image-widths` | - | List raster images without a width in pages that are not generated; `--fix` writes `width="NN%"` from pixels and capture density | +| `image-widths` | - | List raster images that print below 150 ppi in pages that are not generated; `--fix` writes the `width="NN%"` their pixels support in the 16 cm print column | | `localize-bundle-assets` | `task build` | Rewrite the third-party asset URLs Zensical bakes into its JavaScript bundle to the vendored copies | | `update-icons` | `task update:icons` | Fetch the eccenca icon set from the gui-elements repository | | `update-di-reference` | `task update:di-reference` | Generate the task and operator reference pages from a running Corporate Memory | diff --git a/tools/build_pdf.py b/tools/build_pdf.py index e9269b273..c7d32e0a1 100644 --- a/tools/build_pdf.py +++ b/tools/build_pdf.py @@ -1310,10 +1310,12 @@ def load_accepted_low_resolution(path: Path = PRINT_YML) -> set[str]: return set(accepted) -def unaccepted_low_resolution(low_resolution: dict[str, int], accepted: set[str]) -> list[tuple[str, int]]: - """The low-resolution originals that are not accepted, lowest density first.""" +def unaccepted_low_resolution( + low_resolution: dict[str, tuple[int, str]], accepted: set[str] +) -> list[tuple[str, int, str]]: + """The low-resolution originals that are not accepted, lowest density first, each with the width its page declares.""" return sorted( - ((image, ppi) for image, ppi in low_resolution.items() if image not in accepted), + ((image, ppi, width) for image, (ppi, width) in low_resolution.items() if image not in accepted), key=lambda item: (item[1], item[0]), ) @@ -1324,7 +1326,7 @@ def resolve_images( stats: Counter, warnings: list[str], print_images: Path | None = None, - low_resolution: dict[str, int] | None = None, + low_resolution: dict[str, tuple[int, str]] | None = None, ) -> None: """Point images at the files Typst embeds, and replace what it cannot embed. @@ -1335,7 +1337,8 @@ def resolve_images( renders without that text in Typst, so it is reported. With `print_images`, raster images point at their print copies in that directory (`print_image`), and `low_resolution` collects the originals below 150 ppi at their printed - size, by their path under docs/, at their lowest density. + size, by their path under docs/: their lowest density, and the width the + page declares there - the `width="NN%"` of the Markdown source. """ for lightbox in doc.select("a.glightbox"): lightbox.unwrap() @@ -1364,10 +1367,12 @@ def resolve_images( warnings.append(f"SVG text in foreignObject elements does not render, embed a PNG instead: {src}") if print_images is not None and path.suffix.lower() != ".svg": if low_resolution is not None: - density = printed_density(path, img.get("width")) + width = img.get("width", "") + density = printed_density(path, width or None) if density < LOW_RESOLUTION_PPI: source = urllib.parse.unquote(src.split("#")[0].split("?")[0]).lstrip("/") - low_resolution[source] = min(density, low_resolution.get(source, density)) + if source not in low_resolution or density < low_resolution[source][0]: + low_resolution[source] = (density, width) img["src"] = typst_path(print_image(path, img.get("width"), print_images)) stats["images normalized for print"] += 1 else: @@ -1619,7 +1624,7 @@ def normalize( doc: BeautifulSoup, site_dir: Path, print_images: Path | None = None, - low_resolution: dict[str, int] | None = None, + low_resolution: dict[str, tuple[int, str]] | None = None, ) -> tuple[Counter, list[str]]: """Rewrite the merged document into elements pandoc's HTML reader understands. @@ -1826,7 +1831,7 @@ def build_pdf( print(f"Print edition: {count} {'part' if count == 1 else 'parts'} of {md} left out") print(f"Merged {pages} pages and {headings} section headings along {NAV_YML}") - low_resolution: dict[str, int] = {} + low_resolution: dict[str, tuple[int, str]] = {} stats, warnings = normalize(doc, SITE_DIR, work_dir / "images" if print_edition else None, low_resolution) emoji: list[list[str]] = [] if print_edition: @@ -1843,7 +1848,12 @@ def build_pdf( # The originals to replace or accept (backlog P15). unaccepted = unaccepted_low_resolution(low_resolution, load_accepted_low_resolution()) report = work_dir / "low-resolution.tsv" - report.write_text("ppi\timage\n" + "".join(f"{ppi}\t{image}\n" for image, ppi in unaccepted), encoding="utf-8") + # The width is the one the page declares, so the entry says whether the + # image prints too large for its pixels or needs a fresh screenshot. + report.write_text( + "ppi\twidth\timage\n" + "".join(f"{ppi}\t{width}\t{image}\n" for image, ppi, width in unaccepted), + encoding="utf-8", + ) if unaccepted: print( f"Print edition: {len(unaccepted)} images below {LOW_RESOLUTION_PPI} ppi at their printed size, " diff --git a/tools/image_widths.py b/tools/image_widths.py index 76ba6a43e..4cb962d23 100644 --- a/tools/image_widths.py +++ b/tools/image_widths.py @@ -1,15 +1,24 @@ -"""Write the missing width of raster images into the Markdown sources. - -An image without a width prints at the pixel density it declares - 72 dpi when -it declares none - and the site shows it at its pixel size. `dec-tool -image-widths` writes `{ width="NN%" }` instead: the image's natural width as a -share of the full page width. A screenshot then prints at the size it had on the -screen it was captured on, and the site shows it in the same proportion -(tasks/spec.md, §11, D18). Generated pages are left alone: their generators own +"""Size raster images in the Markdown sources so they print sharply. + +An image prints as many pixels per inch as its width on the page allows: a +screenshot of 900 pixels across the full 16 cm column prints at 143 ppi, the +same screenshot at half the column at 286 ppi. `dec-tool image-widths` writes +the `{ width="NN%" }` that brings an image to at least 150 ppi - the density +below which the print edition lists it as too coarse (tasks/spec.md, §11, and +backlog P24). + +The width follows from the pixels alone: `pixels / (16 cm in inches * 150)`, +rounded down, because the declared width cancels out of +`declared * density / target`. It is rounded down to a whole percent, so the +result never falls short of the target, and an image already at 150 ppi or more +keeps the width it has. + +The same width serves the site, where the image then takes the share of the +article it fills on paper. Generated pages are left alone: their generators own them. -Without `--fix` the command lists the images without a width and fails, so a -check can catch a new screenshot; with `--fix` it writes the widths. +Without `--fix` the command lists what prints too coarse and fails, so a check +can catch a new screenshot; with `--fix` it writes the widths. """ from __future__ import annotations @@ -22,41 +31,40 @@ from PIL import Image, UnidentifiedImageError DOCS_DIR = Path("docs") -# The full page width in CSS pixels: the 16 cm text column of the print edition, at 96 per inch. -FULL_PAGE_WIDTH = 16 / 2.54 * 96 +# The text column of the print edition, the width an image at 100 % fills. +COLUMN_INCHES = 16 / 2.54 +TARGET_PPI = 150 RASTER_SUFFIXES = {".png", ".jpg", ".jpeg", ".gif", ".webp"} # The marker a generated page carries (.claude/docs-guidelines/repo-conventions.md). GENERATED_MARK = re.compile(r"generated - do not change it manually|auto-generated", re.IGNORECASE) # A Markdown image, with an optional title, and the attribute list that may follow it. IMAGE = re.compile(r'!\[[^\]]*\]\((?P[^)\s]+)(?:\s+"[^"]*")?\)(?P\{[^}\n]*\})?') +# A width attribute, quoted or not: both spellings occur in the sources. +WIDTH = re.compile(r"""width\s*=\s*["']?(?P\d+)%["']?""") FENCE = re.compile(r"^\s*(```|~~~)") -def capture_scale(dpi: float | None) -> float: - """How many image pixels one CSS pixel took on the screen the image was captured on. +def printed_density(pixels: int, percent: int) -> int: + """How many pixels per inch an image of `pixels` across prints at `percent` of the text column.""" + return round(pixels / (COLUMN_INCHES * percent / 100)) - 144 dpi is a Retina capture on macOS, twice its 72 dpi base. Any other declared - density counts against 96 dpi. An image that declares none, or less than 96 dpi, - has scale 1. - """ - if not dpi or dpi < 96: - return 1.0 - if round(dpi) == 144: - return 2.0 - return dpi / 96 +def target_width(pixels: int, percent: int) -> int: + """The width in percent that brings an image to at least 150 ppi, at most the width it has now. -def width_percent(pixels: int, dpi: float | None) -> int: - """The image's natural width as a share of the full page width, rounded to 5 %, between 5 and 100 %.""" - natural = pixels / capture_scale(dpi) - percent = 5 * math.floor(natural / FULL_PAGE_WIDTH * 20 + 0.5) - return max(5, min(100, percent)) + Rounded down, and a step further where the rounding of the density would + leave it a pixel short of the target. + """ + width = max(1, math.floor(percent * printed_density(pixels, percent) / TARGET_PPI)) + while width > 1 and printed_density(pixels, width) < TARGET_PPI: + width -= 1 + return min(width, percent) -def image_width(page_dir: Path, source: str) -> int | None: - """The width to write for an image a page references, or None for one that gets none. +def image_pixels(page_dir: Path, source: str) -> int | None: + """How many pixels across the image a page references is, or None for one this tool leaves alone. - Remote images, SVGs and files that are missing or unreadable get none. + Remote images, SVGs and files that are missing or unreadable get no width. """ if re.match(r"^[a-z][a-z0-9+.-]*:", source, re.IGNORECASE) or source.startswith("/"): return None @@ -65,51 +73,55 @@ def image_width(page_dir: Path, source: str) -> int | None: return None try: with Image.open(path) as image: - dpi = image.info.get("dpi") - return width_percent(image.width, float(dpi[0]) if dpi and dpi[0] else None) + return image.width except (OSError, UnidentifiedImageError): return None -def add_widths(text: str, page_dir: Path) -> tuple[str, list[tuple[str, int]]]: - """A page's Markdown with a width on every raster image that lacks one, and the images that got one. +def fit_widths(text: str, page_dir: Path) -> tuple[str, list[tuple[str, int, int, int, int]]]: + """A page's Markdown with every image that prints below 150 ppi narrowed, and what changed. - An existing attribute list gains the width; an image without one gets - `{ width="NN%" }`. An image that would get 100 % gets none: it fills the - column in print either way, and the width would only stretch a narrower - screenshot to the article width on the site. Images inside fenced code - blocks stay as they are. + An image without a width fills the column, so it is measured at 100 %. An + existing width is replaced, whether it is quoted or not; an attribute list + without one gains it; an image without attributes gets + `{ width="NN%" }`. Images inside fenced code blocks stay as they are. """ lines = text.split("\n") - changes: list[tuple[str, int]] = [] + changes: list[tuple[str, int, int, int, int]] = [] fenced = False - def widen(match: re.Match) -> str: + def narrow(match: re.Match) -> str: attrs = match.group("attrs") or "" - if "width" in attrs: + declared = WIDTH.search(attrs) + percent = int(declared.group("percent")) if declared else 100 + pixels = image_pixels(page_dir, match.group("src")) + if pixels is None: return match.group(0) - percent = image_width(page_dir, match.group("src")) - if percent is None or percent >= 100: + density = printed_density(pixels, percent) + if density >= TARGET_PPI: return match.group(0) - changes.append((match.group("src"), percent)) + width = target_width(pixels, percent) + changes.append((match.group("src"), percent, width, density, printed_density(pixels, width))) + if declared: + return match.group(0)[: -len(attrs)] + attrs[: declared.start()] + f'width="{width}%"' + attrs[declared.end():] if attrs: - return match.group(0)[: -len(attrs)] + attrs[:-1].rstrip() + f' width="{percent}%" }}' - return match.group(0) + f'{{ width="{percent}%" }}' + return match.group(0)[: -len(attrs)] + attrs[:-1].rstrip() + f' width="{width}%" }}' + return match.group(0) + f'{{ width="{width}%" }}' for index, line in enumerate(lines): if FENCE.match(line): fenced = not fenced continue if not fenced and "![" in line: - lines[index] = IMAGE.sub(widen, line) + lines[index] = IMAGE.sub(narrow, line) return "\n".join(lines), changes @click.command(name="image-widths") -@click.option("--fix", is_flag=True, help="Write the missing widths into the pages instead of listing them.") +@click.option("--fix", is_flag=True, help="Write the widths into the pages instead of listing what is too coarse.") @click.argument("paths", nargs=-1, type=click.Path(exists=True, path_type=Path)) def image_widths(fix: bool, paths: tuple[Path, ...]) -> None: - """List the raster images without a width in pages that are not generated - or write their widths.""" + """List the raster images that print below 150 ppi in pages that are not generated - or narrow them.""" roots = paths or (DOCS_DIR,) pages = sorted({page for root in roots for page in ([root] if root.is_file() else root.rglob("*.md"))}) count = 0 @@ -117,17 +129,19 @@ def image_widths(fix: bool, paths: tuple[Path, ...]) -> None: text = page.read_text(encoding="utf-8") if GENERATED_MARK.search("\n".join(text.splitlines()[:15])): continue - result, changes = add_widths(text, page.parent) + result, changes = fit_widths(text, page.parent) if not changes: continue count += len(changes) - for source, percent in changes: - print(f'{page}: {source} width="{percent}%"') + for source, percent, width, density, target in changes: + print(f"{page}: {source} {percent}% -> {width}%, {density} -> {target} ppi") if fix: page.write_text(result, encoding="utf-8") if fix: print(f"{count} image widths written") elif count: - raise click.ClickException(f"{count} images have no width - run `dec-tool image-widths --fix`") + raise click.ClickException( + f"{count} images print below {TARGET_PPI} ppi - run `dec-tool image-widths --fix`" + ) else: - print("no image needs a width") + print(f"every image prints at {TARGET_PPI} ppi or more") From 1f5e04286d8a6ade9fd7b9ddf53f18eecd799490 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Wed, 16 Sep 2026 14:29:54 +0200 Subject: [PATCH 09/15] share the print geometry between the PDF tools MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The text column and the two densities were defined again in each tool, so the report's threshold and the width tool's target could drift apart, and two functions named printed_density measured different things. - move the 16 cm column and the 300/150 ppi densities to tools/print_geometry.py, and import them in the builder, the width tool and the preflight - rename build_pdf.printed_density to typst_density: it mirrors Typst's sizing, while the width tool measures a share of the column - take FOGRA39 from pdf_normalize instead of repeating it in the preflight - run image-widths in `task check` as check:images - correct the status lines, spec §8 and the print.yml comment, which still said a coarse image is replaced rather than narrowed --- Taskfile.yml | 13 +++++++++++++ tasks/backlog.md | 6 +++--- tasks/spec.md | 14 ++++++++------ tools/README.md | 2 +- tools/build_pdf.py | 20 ++++++++++++-------- tools/image_widths.py | 8 +++++--- tools/pdf/print.yml | 7 ++++--- tools/pdf_preflight.py | 6 ++---- tools/print_geometry.py | 20 ++++++++++++++++++++ 9 files changed, 68 insertions(+), 28 deletions(-) create mode 100644 tools/print_geometry.py diff --git a/Taskfile.yml b/Taskfile.yml index fc24bed24..4a4b6c45a 100644 --- a/Taskfile.yml +++ b/Taskfile.yml @@ -69,6 +69,7 @@ tasks: cmds: - task: check:links - task: check:rumdl + - task: check:images - task: check:yamllint - task: check:navigation - task: check:output @@ -82,6 +83,18 @@ tasks: # CI run agree: GitHub runners have no user-level yamllint config. - poetry run yamllint . + check:images: + desc: Fail if a screenshot prints below 150 ppi in the print edition + summary: | + Runs `dec-tool image-widths`: every raster image in a page that is not + generated must carry a width small enough for its pixels, or it prints + coarse (tasks/spec.md, §11, D18). `dec-tool image-widths --fix` writes + the width a listed image needs. + deps: + - install + cmds: + - poetry run dec-tool image-widths + check:navigation: desc: Fail if nav.yml is out of sync with the docs/**/.pages files deps: diff --git a/tasks/backlog.md b/tasks/backlog.md index 69fa3cb1c..35135c9ad 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -1,7 +1,7 @@ # Backlog: print-on-demand book block -Work breakdown for [spec.md](spec.md). **Status 2026-09-15: P0-P14 and P18-P24 done and verified; P15 has its tooling and -42 images to replace or accept; P16 and P17 open.** +Work breakdown for [spec.md](spec.md). **Status 2026-09-16: P0-P15 and P18-P25 done and verified; +P16 (CI) and P17 (the cover) open.** The previous content of this file (the temporary tag-listing renderer) is in the git history. @@ -611,7 +611,7 @@ missing width attributes. **Superseded:** the first implementation (2026-09-15, D18 as written) took the natural width from the capture scale, rounded it to 5 % and skipped anything at 100 %. Its 24 widths stay where the new rule does -not narrow them further. `task check` does not run the check yet. +not narrow them further. `task check` runs the check as `check:images`. --- diff --git a/tasks/spec.md b/tasks/spec.md index b511cf546..fd80a4a86 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -1,13 +1,15 @@ # Spec: print-on-demand book block -**Status:** accepted 2026-09-14 - all decisions made (§4). Backlog P0-P12 implemented 2026-09-15 -(`task pdf:print`, 962 pages); P13-P17 open. The backlog's "Done" notes record where the implementation +**Status:** accepted 2026-09-14 - all decisions made (§4). Backlog P0-P15 and P18-P25 are implemented; +P16 (CI) and P17 (the cover) are open. The backlog's "Done" notes record where the implementation refines this spec. §10, excluding content from the print edition, was decided on 2026-09-15 (D12-D14) and implemented the same day (backlog P18): the print edition has 666 pages. §11 collects the pull request review of 2026-09-15, decided the same day (D15-D18); backlog P19-P24, of which P19-P23 were implemented on 2026-09-15: the print edition has 870 pages. P13, P14 and P24, and the tooling of P15, followed the same day: PDF/X-4 normalization (§7, with the transparency Ghostscript cannot convert rendered to images), the -preflight report, image widths in the sources and the list of low-resolution originals; 868 pages. +preflight report, image widths in the sources and the list of low-resolution originals. On 2026-09-16 the +width rule was revised (D18, P24, which closed P15) and a short lead line under a heading became sticky +(P25): the print edition has 864 pages. **Branch:** `feature/print-on-demand`, based on `main` at `c20d74b94` (PDF export merged). **Goal:** a *book block* - the interior file of a printed, perfect-bound book - built next to the screen PDF, which BoD accepts without rework. @@ -408,8 +410,8 @@ condition is not a registered one. `--icc-profile` names another grey profile. - page count even and at most 1,200 - grey areas at least 20 % black; all fonts embedded; no soft masks left - every image at 300 ppi or more at its printed size; an original below 150 ppi is listed in - `dist/pdf/print/low-resolution.tsv` until it is replaced or accepted under `accepted-low-resolution` in - `print.yml` + `dist/pdf/print/low-resolution.tsv`, which `dec-tool image-widths` empties by narrowing the image to the + width its pixels support - `accepted-low-resolution` in `print.yml` keeps one at its size instead - with normalization: the file declares PDF/X-4, carries a FOGRA39 output intent, CMYK images only, no image above 300 ppi - a page key in `print.yml` drops that page; `omit` leaves no title and no note for a page or a section @@ -746,7 +748,7 @@ with `--fix`, narrows every raster image that prints below 150 ppi in pages that stay as they are, as do generated pages. - **No floor:** an image whose pixels only support a thumbnail gets the thumbnail; the two smallest are 14 % and 21 %, each a snippet of a dialog. -- **Check:** `task check` may run the check, so a new screenshot is caught. +- **Check:** `task check` runs it as `check:images`, so a new screenshot is caught before it is built. The same percentage serves the site and the PDF: on paper the image prints at 150 ppi or better, and the site shows it at the share of the article it fills on the page. This closes P15 - `--fix` wrote 47 widths diff --git a/tools/README.md b/tools/README.md index 453649137..234a8cbce 100644 --- a/tools/README.md +++ b/tools/README.md @@ -15,7 +15,7 @@ supported entry points and pass the right options. | `pdf-normalize` | `task pdf:print -- --normalize`, `task pdf:print -- --gray` | Write the print PDF as PDF/X-4 in CMYK with Ghostscript, checking the result for the PDF/X marker; the ISO Coated v2 profile is fetched into `dist/icc/`. `--gray` writes a greyscale preview of the black-and-white print instead, PDF/X-4 by the grey profile Ghostscript ships | | `pdf-preflight` | `task pdf:print` | Check the book block for print: A4, even page count, embedded fonts, image resolution, soft masks, annotations, page-number position, blank pages, light grey areas | | `check-zensical-output` | `task check:output` | Inspect the built `site/` and fail if a feature we reimplemented for Zensical regressed | -| `image-widths` | - | List raster images that print below 150 ppi in pages that are not generated; `--fix` writes the `width="NN%"` their pixels support in the 16 cm print column | +| `image-widths` | `task check:images` | List raster images that print below 150 ppi in pages that are not generated; `--fix` writes the `width="NN%"` their pixels support in the 16 cm print column | | `localize-bundle-assets` | `task build` | Rewrite the third-party asset URLs Zensical bakes into its JavaScript bundle to the vendored copies | | `update-icons` | `task update:icons` | Fetch the eccenca icon set from the gui-elements repository | | `update-di-reference` | `task update:di-reference` | Generate the task and operator reference pages from a running Corporate Memory | diff --git a/tools/build_pdf.py b/tools/build_pdf.py index c7d32e0a1..a93184c82 100644 --- a/tools/build_pdf.py +++ b/tools/build_pdf.py @@ -45,6 +45,7 @@ from tools.pdf_authors import load_imprint_names from tools.pdf_normalize import ensure_profile, gray_path, gray_preview, normalize as normalize_pdf, normalized_path from tools.pdf_preflight import print_report, run_preflight +from tools.print_geometry import COLUMN_INCHES, LOW_RESOLUTION_PPI, PRINT_PPI, TEXT_WIDTH_PT DEFAULT_OUT_STEM = "dist/documentation-eccenca-com" SITE_DIR = Path("site") @@ -116,10 +117,8 @@ OPERATOR_SECTIONS_LEFT_OUT = {"examples", "example", "parameter", "parameters", "advanced parameter", "related plugins"} # The print edition's text column is 16 cm wide, and BoD asks for images at # 300 dpi at their printed size (tasks/spec.md, §2). -TEXT_WIDTH_PT = 16 / 2.54 * 72 -PRINT_PPI = 300 -# Originals below this density at their printed size need replacing or accepting (backlog P15). -LOW_RESOLUTION_PPI = 150 +# The column and the densities come from tools/print_geometry.py, which the +# image tools and the preflight share. # Colour emoji in the print edition: rendered as images from the vendored font, # for text the body font does not cover. EMOJI_FONT = PDF_ASSETS / "fonts" / "noto-color-emoji" / "Noto-COLRv1.ttf" @@ -127,7 +126,7 @@ # What the print edition renders to PNG with Typst: colour emoji and SVGs that # draw with transparency - opacity, masks, filters, translucent colours. RENDER_PPI = 600 -MAX_RENDER_WIDTH = round(16 / 2.54 * RENDER_PPI) +MAX_RENDER_WIDTH = round(COLUMN_INCHES * RENDER_PPI) SVG_TRANSPARENCY = re.compile(r"opacity| Path: return target -def printed_density(source: Path, width: str | None) -> int: - """An original image's pixel density at the width the print edition prints it, in ppi.""" +def typst_density(source: Path, width: str | None) -> int: + """An original image's pixel density at the width Typst prints it, in ppi. + + Typst sizes an image by the width the page declares, else by the density the + file declares, capped at the column - `image_widths.printed_density` instead + measures a share of the column, which is what a source may declare. + """ with Image.open(source) as image: return round(image.width / (printed_width_pt(image, width) / 72)) @@ -1368,7 +1372,7 @@ def resolve_images( if print_images is not None and path.suffix.lower() != ".svg": if low_resolution is not None: width = img.get("width", "") - density = printed_density(path, width or None) + density = typst_density(path, width or None) if density < LOW_RESOLUTION_PPI: source = urllib.parse.unquote(src.split("#")[0].split("?")[0]).lstrip("/") if source not in low_resolution or density < low_resolution[source][0]: diff --git a/tools/image_widths.py b/tools/image_widths.py index 4cb962d23..20f9b5551 100644 --- a/tools/image_widths.py +++ b/tools/image_widths.py @@ -30,10 +30,12 @@ import click from PIL import Image, UnidentifiedImageError +from tools.print_geometry import COLUMN_INCHES, LOW_RESOLUTION_PPI + DOCS_DIR = Path("docs") -# The text column of the print edition, the width an image at 100 % fills. -COLUMN_INCHES = 16 / 2.54 -TARGET_PPI = 150 +# The text column an image at 100 % fills, and the density it must reach, both +# shared with the builder and the preflight (tools/print_geometry.py). +TARGET_PPI = LOW_RESOLUTION_PPI RASTER_SUFFIXES = {".png", ".jpg", ".jpeg", ".gif", ".webp"} # The marker a generated page carries (.claude/docs-guidelines/repo-conventions.md). GENERATED_MARK = re.compile(r"generated - do not change it manually|auto-generated", re.IGNORECASE) diff --git a/tools/pdf/print.yml b/tools/pdf/print.yml index 36e025842..8cdccb084 100644 --- a/tools/pdf/print.yml +++ b/tools/pdf/print.yml @@ -50,9 +50,10 @@ authors: # darker in black and white. body-weight: light -# Images below 150 ppi at their printed size that are accepted as they are, by -# their path under docs/. The print build lists the others in -# dist/pdf/print/low-resolution.tsv, to replace (backlog P15). +# Images below 150 ppi at their printed size that are accepted at that size, by +# their path under docs/. `dec-tool image-widths --fix` narrows such an image +# instead, which keeps this list empty; what is left over stands in +# dist/pdf/print/low-resolution.tsv (backlog P15, P24). accepted-low-resolution: [] # How a navigation section or a page is printed, keyed by the docs/ directory diff --git a/tools/pdf_preflight.py b/tools/pdf_preflight.py index 2e65a28e7..1df899af4 100644 --- a/tools/pdf_preflight.py +++ b/tools/pdf_preflight.py @@ -22,14 +22,12 @@ import click from pypdf import PdfReader -from tools.pdf_normalize import is_pdfx4 +from tools.pdf_normalize import FOGRA39, is_pdfx4 +from tools.print_geometry import PRINT_PPI as MIN_PPI A4 = (595.28, 841.89) -# The output intent of the book block; the greyscale preview names its grey profile instead. -FOGRA39 = "FOGRA39" SIZE_TOLERANCE = 1.0 MAX_PAGES = 1200 -MIN_PPI = 300 # pdfimages rounds densities to whole pixels. PPI_TOLERANCE = 1 MIN_AREA_BLACK = 0.20 diff --git a/tools/print_geometry.py b/tools/print_geometry.py new file mode 100644 index 000000000..670e32eed --- /dev/null +++ b/tools/print_geometry.py @@ -0,0 +1,20 @@ +"""What the print edition measures images by: the text column and two densities. + +The width of the column decides how sharp an image prints, so the builder +(`build_pdf`), the tool that sizes images in the sources (`image_widths`) and +the preflight report (`pdf_preflight`) have to agree on it. They share these +values rather than each keeping their own (tasks/spec.md, §5 and §11). +""" +from __future__ import annotations + +# The text column of the A4 page, in the three units the tools work in. +COLUMN_CM = 16 +COLUMN_INCHES = COLUMN_CM / 2.54 +TEXT_WIDTH_PT = COLUMN_INCHES * 72 + +# What print asks for: images are resampled to 300 ppi at their printed width +# (BoD, tasks/spec.md, §6), and the preflight fails below it. +PRINT_PPI = 300 +# Below this density an original is too coarse to print at that size: the build +# lists it and `dec-tool image-widths` narrows it (backlog P15, P24). +LOW_RESOLUTION_PPI = 150 From 132d6691ad629de031d345c2ff8277a8af32dafa Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Wed, 16 Sep 2026 14:56:52 +0200 Subject: [PATCH 10/15] print the first edition as a user guide MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first print edition serves users, so what an administrator or a developer needs stays in the online edition. - omit part E (deploy-and-configure/, 34 pages) and part G (develop/, 89 pages) in print.yml; develop/cmem-client-api/ goes with it, since section keys do not nest - the book drops from 864 to 668 pages, and the parts re-letter from E Automate onwards - a reference into an omitted part becomes a web address in its part's endnote list, so none is lost - record the scope in spec §10 and backlog P26 --- tasks/backlog.md | 27 +++++++++++++++++++++++---- tasks/spec.md | 13 ++++++++++--- tools/pdf/print.yml | 5 ++++- 3 files changed, 37 insertions(+), 8 deletions(-) diff --git a/tasks/backlog.md b/tasks/backlog.md index 35135c9ad..057f09cfb 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -1,6 +1,6 @@ # Backlog: print-on-demand book block -Work breakdown for [spec.md](spec.md). **Status 2026-09-16: P0-P15 and P18-P25 done and verified; +Work breakdown for [spec.md](spec.md). **Status 2026-09-16: P0-P15 and P18-P26 done and verified; P16 (CI) and P17 (the cover) open.** The previous content of this file (the temporary tag-listing renderer) is in the git history. @@ -414,7 +414,8 @@ Spec §10, D12-D14: leave subtrees, pages and parts of a page out of the print e - the site and the screen PDF ignore the class - documentation: the `sections` comment in `print.yml`, and a line for authors in `.claude/docs-guidelines/repo-conventions.md` on the class, which generated pages cannot carry -- configuration: `develop/cmem-client-api/: omit`, `build/tutorial-how-to-link-ids-to-osint/: omit` (all +- configuration: `develop/cmem-client-api/: omit` (P26 widened it to all of `develop/`), + `build/tutorial-how-to-link-ids-to-osint/: omit` (all 7 pages), and `{ .sql .print-exclude }` on the SQL code block of the Snowflake tutorial - lines 92-1094 of `docs/build/snowflake-tutorial/index.md` on 2026-09-15, inside `??? example "INSERT query"`, which keeps its title @@ -436,8 +437,8 @@ from the stamp. `task check` passes. - `exclude_parts` replaces each run of `.print-exclude` elements with the note, whose address carries the anchor of the heading before the part. `merge_pages` returns the parts left out per page, which the build logs. List tables skip marked elements as well. -- `print.yml` omits `develop/cmem-client-api/` (75 pages) and `build/tutorial-how-to-link-ids-to-osint/` - (7 pages). The SQL block of the Snowflake tutorial carries `sql { .print-exclude }`. Authors find the +- `print.yml` omits `develop/cmem-client-api/` (75 pages, widened to all of `develop/` by P26) and + `build/tutorial-how-to-link-ids-to-osint/` (7 pages). The SQL block of the Snowflake tutorial carries `sql { .print-exclude }`. Authors find the markup in `.claude/docs-guidelines/repo-conventions.md`. - Measured: - The print edition drops from 966 to 666 pages, the Snowflake tutorial from 46 to 22. @@ -635,6 +636,24 @@ change if the shorter book matters more. The screen edition does not run the ste --- +## P26 - The first edition is a user guide - **done** + +Decided 2026-09-16: the first print edition serves users, so what an administrator or a developer needs +stays in the online edition. `tools/pdf/print.yml` omits part E (`deploy-and-configure/`, 34 pages) and +part G (`develop/`, 89 pages) whole, and the narrower `develop/cmem-client-api/` key goes, since section +keys do not nest. + +**Verified:** the book drops from 864 to 668 pages and the preflight passes unchanged. No cross-reference +dies: a link from a printed page into an omitted part becomes a web address in its part's endnote list - +the JDBC driver setup, for one, prints as +`https://documentation.eccenca.com/latest/deploy-and-configure/configuration/dataintegration/jdbc/` - +and internal links without a target fall from 17 to 4, because most of those lived in the dropped trees. + +**Open:** the imprint's author list counts commits to the printed pages (D16), and 123 pages left the +book, so `task pdf:authors` should run before the edition goes to print. + +--- + ## Sequencing ```text diff --git a/tasks/spec.md b/tasks/spec.md index fd80a4a86..4edb66bf1 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -1,6 +1,6 @@ # Spec: print-on-demand book block -**Status:** accepted 2026-09-14 - all decisions made (§4). Backlog P0-P15 and P18-P25 are implemented; +**Status:** accepted 2026-09-14 - all decisions made (§4). Backlog P0-P15 and P18-P26 are implemented; P16 (CI) and P17 (the cover) are open. The backlog's "Done" notes record where the implementation refines this spec. §10, excluding content from the print edition, was decided on 2026-09-15 (D12-D14) and implemented the same day (backlog P18): the print edition has 666 pages. §11 collects the pull request @@ -9,7 +9,8 @@ on 2026-09-15: the print edition has 870 pages. P13, P14 and P24, and the toolin day: PDF/X-4 normalization (§7, with the transparency Ghostscript cannot convert rendered to images), the preflight report, image widths in the sources and the list of low-resolution originals. On 2026-09-16 the width rule was revised (D18, P24, which closed P15) and a short lead line under a heading became sticky -(P25): the print edition has 864 pages. +(P25). On 2026-09-16 the first edition's scope was set to a user guide, omitting parts E and G whole +(P26): the print edition has 668 pages. **Branch:** `feature/print-on-demand`, based on `main` at `c20d74b94` (PDF export merged). **Goal:** a *book block* - the interior file of a printed, perfect-bound book - built next to the screen PDF, which BoD accepts without rework. @@ -482,7 +483,8 @@ A key under `sections` names a docs directory, ending in `/` as today, or a sing ```yaml sections: - develop/cmem-client-api/: omit + deploy-and-configure/: omit + develop/: omit build/tutorial-how-to-link-ids-to-osint/: omit # a single page: /index.md: omit ``` @@ -493,6 +495,11 @@ sections: - A page key accepts `omit` only; any other mode fails the build. The page is dropped without a note. - Dropping the index page of a section keeps its other pages, under a heading with the section's navigation title - the shape Release Notes already has (§3). +- **Scope of the first edition (decided 2026-09-16, backlog P26):** it is a user guide, so the parts an + administrator or a developer needs are omitted whole - E `deploy-and-configure/` (34 pages) and + G `develop/` (89 pages). The narrower `develop/cmem-client-api/` key goes with it, because keys do not + nest. A reference from a printed page into an omitted part is not lost: it becomes a web address in its + part's endnote list (D15), pointing at the online edition. - Keys do not nest: a key inside a directory that another key shortens fails the build. - A key that matches no page in `nav.yml` fails the build, as today. - A link to a dropped page prints the page's online address in a footnote, like any link that leaves diff --git a/tools/pdf/print.yml b/tools/pdf/print.yml index 8cdccb084..dc838b9dc 100644 --- a/tools/pdf/print.yml +++ b/tools/pdf/print.yml @@ -77,5 +77,8 @@ sections: release-notes/: mode: list columns: [Release, Summary] - develop/cmem-client-api/: omit + # The first print edition is a user guide: what an administrator or a + # developer needs stays in the online edition (parts E and G). + deploy-and-configure/: omit + develop/: omit build/tutorial-how-to-link-ids-to-osint/: omit From aa6df67c1ef5e37e4985031647e6b39e98ee59b3 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Wed, 16 Sep 2026 15:08:20 +0200 Subject: [PATCH 11/15] leave the release notes out of the print edition The user guide does not carry a release history; the online edition has it. The two-column list configuration stays as a comment, so switching back is possible. --- tools/pdf/print.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tools/pdf/print.yml b/tools/pdf/print.yml index dc838b9dc..502559473 100644 --- a/tools/pdf/print.yml +++ b/tools/pdf/print.yml @@ -74,9 +74,9 @@ accepted-low-resolution: [] # e.g. ```sql { .print-exclude }; a note naming the page online takes its place. sections: build/reference/: reference - release-notes/: - mode: list - columns: [Release, Summary] + release-notes/: omit + # mode: list + # columns: [Release, Summary] # The first print edition is a user guide: what an administrator or a # developer needs stays in the online edition (parts E and G). deploy-and-configure/: omit From 52b9cc032e5c4f7eedfd4f7c49023443be6fb185 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Wed, 16 Sep 2026 15:25:43 +0200 Subject: [PATCH 12/15] declare the dependencies the tools import CI installs from the lock file and had no pypdf, so every test module failed to import: tools/__init__.py loads build_pdf, which loads pdf_normalize. Locally it worked because pypdf sat in the virtualenv from an install outside poetry. - declare pypdf for the PDF reader of pdf_normalize and pdf_preflight - declare click, markdown and requests too: the tools import them directly, but they only arrived as dependencies of other packages --- poetry.lock | 24 +++++++++++++++++++++++- pyproject.toml | 12 ++++++++++++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/poetry.lock b/poetry.lock index 1df274afd..a1ed0239c 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1682,6 +1682,28 @@ files = [ [package.extras] diagrams = ["jinja2", "railroad-diagrams"] +[[package]] +name = "pypdf" +version = "6.19.0" +description = "A pure-python PDF library capable of splitting, merging, cropping, and transforming PDF files" +optional = false +python-versions = ">=3.9" +groups = ["main"] +files = [ + {file = "pypdf-6.19.0-py3-none-any.whl", hash = "sha256:7e5d6e730e7dae87d560a2cee218b852f6498c8be61966f3cd02ead971e48d14"}, + {file = "pypdf-6.19.0.tar.gz", hash = "sha256:bbc43aca292369ccc6cbc8a921991ecf2538a3587ab5a116eff06c321d647155"}, +] + +[package.extras] +crypto = ["cryptography (>3.0)"] +cryptodome = ["PyCryptodome"] +dev = ["flit", "pip-tools", "pre-commit", "pytest-cov", "pytest-socket", "pytest-timeout", "pytest-xdist", "wheel"] +docs = ["myst_parser", "sphinx", "sphinx_rtd_theme"] +fonts = ["fonttools"] +full = ["Pillow (>=8.0.0)", "arabic-reshaper", "cryptography (>3.0)", "fonttools", "python-bidi"] +image = ["Pillow (>=8.0.0)"] +rtl-text = ["arabic-reshaper", "python-bidi"] + [[package]] name = "pytest" version = "9.1.1" @@ -2252,4 +2274,4 @@ tomli = ">=2.4.0" [metadata] lock-version = "2.1" python-versions = "^3.11" -content-hash = "11d58954fa95ccfda69482e9b9f98c6b353b358999f725966da0fa5b9459f9dc" +content-hash = "f44cab4e2c2174d1399b9eb7e209f2bc26ff6af9b3cd16995a622304a606d327" diff --git a/pyproject.toml b/pyproject.toml index efd06901b..79d0e1298 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -41,6 +41,18 @@ jinja2 = "^3.1.6" # `task build` plus `tools/check_zensical_output.py` when doing so. zensical = "^0.0.62" pillow = "^11.3.0" +# tools/pdf_normalize.py and tools/pdf_preflight.py read the PDFs the print +# edition writes: the PDF/X-4 marker, the output intent, page sizes and annotations. +pypdf = "^6.18" +# Every dec-tool subcommand is a click command (tools/__init__.py). click, markdown +# and requests used to arrive as dependencies of other packages; a tool that imports +# them directly declares them, or an upstream bump takes them away. +click = "^8.5" +# tools/build_pdf.py renders the Markdown fields of data/plugins.json for the +# operator reference of the print edition. +markdown = "^3.10" +# tools/update_icons.py fetches the eccenca icon set from the gui-elements repository. +requests = "^2.32" [tool.poetry.group.dev.dependencies] linkcheckmd = "^1.4.0" From 80ccd1897a74717ee9468ef96777fe882585baf9 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Wed, 16 Sep 2026 15:46:00 +0200 Subject: [PATCH 13/15] build every pdf edition in CI and link it in the run summary The PDF workflow built only the screen edition, on main, and its artifact had to be dug out of the run page. - build the book block, its PDF/X-4 copy and the greyscale preview too, on main and on the print-on-demand branch, or on demand - install ghostscript and poppler-utils: the preflight report reads the PDFs with pdffonts, pdfimages and pdftotext - upload one artifact per edition and list them in the run summary with page count, size and download link - cache the output intent profile, and pass ghostscript's grey profile as PDF_GRAY_PROFILE, which ensure_gray_profile now honours - give each ref its own concurrency group, so main and a branch no longer cancel each other --- .github/workflows/pdf.yml | 115 ++++++++++++++++++++++++++++++++---- tasks/backlog.md | 29 ++++++--- tasks/spec.md | 4 +- tests/test_pdf_normalize.py | 10 ++++ tools/pdf_normalize.py | 8 ++- 5 files changed, 145 insertions(+), 21 deletions(-) diff --git a/.github/workflows/pdf.yml b/.github/workflows/pdf.yml index 61c481a27..c3e6228ba 100644 --- a/.github/workflows/pdf.yml +++ b/.github/workflows/pdf.yml @@ -1,18 +1,24 @@ --- name: PDF -# Builds the PDF of the whole documentation after every push to main, merged -# pull requests included, and keeps it as an artifact of the workflow run. +# Builds every PDF of the documentation after a push to main or to the +# print-on-demand branch: the screen edition, the print edition (the book block +# for print on demand), its PDF/X-4 copy in CMYK and the greyscale preview. The +# run summary lists all four with their download links, so they need not be dug +# out of the artifact section. on: push: - branches: ["main"] + branches: ["main", "feature/print-on-demand"] + # Allows you to run this workflow manually from the Actions tab + workflow_dispatch: permissions: contents: read -# A newer push to main makes a running build obsolete. +# A newer push to the same branch makes a running build obsolete; main and a +# branch do not cancel each other. concurrency: - group: "pdf" + group: "pdf-${{ github.ref }}" cancel-in-progress: true env: @@ -62,8 +68,10 @@ jobs: - name: update packages run: sudo apt-get update -y + # Ghostscript converts the book block to PDF/X-4 and to greyscale; poppler + # carries the pdffonts, pdfimages and pdftotext the preflight report reads. - name: install packages - run: sudo apt-get install -y libffi-dev libz-dev + run: sudo apt-get install -y libffi-dev libz-dev ghostscript poppler-utils - name: install pandoc run: | @@ -79,14 +87,101 @@ jobs: | tar -xJ -C "$RUNNER_TEMP" sudo install -m 0755 "$RUNNER_TEMP/typst-x86_64-unknown-linux-musl/typst" /usr/local/bin/typst - - name: build pdf + # The ISO Coated v2 profile of the output intent is not in the repository + # (its licence forbids it), so it is fetched once and kept between runs. + - name: cache the output intent profile + uses: actions/cache@v4 + with: + path: dist/icc + key: icc-iso-coated-v2-eci + + # Where a distribution puts Ghostscript's own grey profile differs, so the + # greyscale pass is told which file to use instead of guessing. + - name: find the grey profile + run: | + profile="$(find /usr/share /usr/lib -name default_gray.icc -print -quit 2>/dev/null || true)" + if [ -z "$profile" ]; then + echo "no default_gray.icc found - install it or drop --gray" >&2 + exit 1 + fi + echo "PDF_GRAY_PROFILE=$profile" >> "$GITHUB_ENV" + + - name: build the screen pdf run: task pdf - - name: upload pdf + - name: build the print pdfs + run: task pdf:print -- --normalize --gray + + - name: upload the screen pdf + id: screen uses: actions/upload-artifact@v4 with: - name: documentation-eccenca-com-pdf - path: dist/documentation-eccenca-com-*.pdf + name: pdf-screen + path: | + dist/documentation-eccenca-com-*.pdf + !dist/documentation-eccenca-com-*-print*.pdf if-no-files-found: error + retention-days: 30 # A PDF is compressed already. compression-level: 0 + + - name: upload the print pdf + id: print + uses: actions/upload-artifact@v4 + with: + name: pdf-print + path: dist/documentation-eccenca-com-*-print.pdf + if-no-files-found: error + retention-days: 30 + compression-level: 0 + + - name: upload the pdf/x-4 pdf + id: pdfx + uses: actions/upload-artifact@v4 + with: + name: pdf-print-x4 + path: dist/documentation-eccenca-com-*-print-x4.pdf + if-no-files-found: error + retention-days: 30 + compression-level: 0 + + - name: upload the greyscale pdf + id: gray + uses: actions/upload-artifact@v4 + with: + name: pdf-print-gray-x4 + path: dist/documentation-eccenca-com-*-print-gray-x4.pdf + if-no-files-found: error + retention-days: 30 + compression-level: 0 + + - name: list the pdfs in the run summary + run: | + row() { + file="$1" + edition="$2" + url="$3" + pages="$(pdfinfo "$file" | awk '/^Pages/ {print $2}')" + size="$(du -h "$file" | cut -f1)" + printf '| %s | %s | %s | [%s](%s) |\n' \ + "$edition" "$pages" "$size" "$(basename "$file")" "$url" >> "$GITHUB_STEP_SUMMARY" + } + { + echo "## PDF editions of ${GITHUB_REF_NAME}" + echo + echo "| Edition | Pages | Size | Download |" + echo "| :------ | ----: | ---: | :------- |" + } >> "$GITHUB_STEP_SUMMARY" + row "$(ls dist/documentation-eccenca-com-*.pdf | grep -v -- '-print' | head -1)" \ + "Screen" "${{ steps.screen.outputs.artifact-url }}" + row "$(ls dist/documentation-eccenca-com-*-print.pdf | head -1)" \ + "Print, book block" "${{ steps.print.outputs.artifact-url }}" + row "$(ls dist/documentation-eccenca-com-*-print-x4.pdf | head -1)" \ + "Print, PDF/X-4 in CMYK" "${{ steps.pdfx.outputs.artifact-url }}" + row "$(ls dist/documentation-eccenca-com-*-print-gray-x4.pdf | head -1)" \ + "Print, greyscale preview" "${{ steps.gray.outputs.artifact-url }}" + { + echo + echo "The print editions carry the preflight report of \`dec-tool pdf-preflight\`;" + echo "a failed check fails this job." + } >> "$GITHUB_STEP_SUMMARY" diff --git a/tasks/backlog.md b/tasks/backlog.md index 057f09cfb..d9d5a1ae1 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -1,7 +1,7 @@ # Backlog: print-on-demand book block -Work breakdown for [spec.md](spec.md). **Status 2026-09-16: P0-P15 and P18-P26 done and verified; -P16 (CI) and P17 (the cover) open.** +Work breakdown for [spec.md](spec.md). **Status 2026-09-16: P0-P16 and P18-P26 done and verified; +P17 (the cover) open.** The previous content of this file (the temporary tag-listing renderer) is in the git history. @@ -378,12 +378,25 @@ the report is empty. --- -## P16 - CI - -`.github/workflows/pdf.yml` also builds the normalized book block and uploads it with the preflight -report, as a separate artifact. Installs Ghostscript on the runner. - -**Verify:** a push to `main` produces both artifacts. +## P16 - CI - **done** + +Done 2026-09-16: `.github/workflows/pdf.yml` builds every edition after a push to `main` or to +`feature/print-on-demand`, and on demand from the Actions tab - the screen PDF from `task pdf`, and the +book block, its PDF/X-4 copy in CMYK and the greyscale preview from `task pdf:print -- --normalize --gray`. + +- **The runner gets Ghostscript and poppler.** The preflight report shells out to `pdffonts`, `pdfimages` + and `pdftotext`, so without poppler the print build fails rather than the check being skipped. +- **Four artifacts**, one per edition (`pdf-screen`, `pdf-print`, `pdf-print-x4`, `pdf-print-gray-x4`), + kept 30 days and stored uncompressed, since a PDF is compressed already. +- **The run summary** lists each edition with its page count, size and download link, so nothing has to be + dug out of the artifact section (user decision 2026-09-16: summary links, no release page). +- **The output intent profile** is cached in `dist/icc` between runs; its licence keeps it out of the + repository, and CI would otherwise fetch it from the ECI every time. +- **The grey profile** is located with `find` and passed as `PDF_GRAY_PROFILE`, because distributions put + Ghostscript's `default_gray.icc` in different places; `ensure_gray_profile` honours that variable. +- Concurrency is per ref, so a push to `main` no longer cancels a branch build. + +**Verify:** a push produces the four artifacts and the summary table; a failing preflight fails the job. **Est:** small. **Depends on:** P1, P13, P14. --- diff --git a/tasks/spec.md b/tasks/spec.md index 4edb66bf1..563bc259d 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -1,7 +1,7 @@ # Spec: print-on-demand book block -**Status:** accepted 2026-09-14 - all decisions made (§4). Backlog P0-P15 and P18-P26 are implemented; -P16 (CI) and P17 (the cover) are open. The backlog's "Done" notes record where the implementation +**Status:** accepted 2026-09-14 - all decisions made (§4). Backlog P0-P16 and P18-P26 are implemented; +P17 (the cover) is open. The backlog's "Done" notes record where the implementation refines this spec. §10, excluding content from the print edition, was decided on 2026-09-15 (D12-D14) and implemented the same day (backlog P18): the print edition has 666 pages. §11 collects the pull request review of 2026-09-15, decided the same day (D15-D18); backlog P19-P24, of which P19-P23 were implemented diff --git a/tests/test_pdf_normalize.py b/tests/test_pdf_normalize.py index c3aeff897..3cd0cde01 100644 --- a/tests/test_pdf_normalize.py +++ b/tests/test_pdf_normalize.py @@ -70,6 +70,16 @@ def test_each_output_is_named_after_what_it_is(): assert gray_path(book).name == "documentation-eccenca-com-26-2-print-gray-x4.pdf" +def test_the_grey_profile_can_be_named_by_the_environment(tmp_path, monkeypatch): + profile = tmp_path / "gray.icc" + profile.write_bytes(b"icc profile bytes") + monkeypatch.setenv("PDF_GRAY_PROFILE", str(profile)) + assert ensure_gray_profile() == profile + monkeypatch.setenv("PDF_GRAY_PROFILE", str(tmp_path / "gone.icc")) + with pytest.raises(click.ClickException, match="not found"): + ensure_gray_profile() + + def test_the_grey_prefix_names_one_component_and_no_registry(): prefix = pdfx_prefix( Path("/profiles/gray.icc"), "Book", components=1, identifier="sGray", condition="A grey profile", diff --git a/tools/pdf_normalize.py b/tools/pdf_normalize.py index 930e5d092..6455dd6d5 100644 --- a/tools/pdf_normalize.py +++ b/tools/pdf_normalize.py @@ -177,7 +177,13 @@ def ensure_profile(profile: Path | None = None) -> Path: def ensure_gray_profile(profile: Path | None = None) -> Path: - """The grey output intent profile: the one named, else the generic one Ghostscript ships.""" + """The grey output intent profile: the one named, else PDF_GRAY_PROFILE, else the one Ghostscript ships. + + Where a distribution puts `default_gray.icc` differs, so a build that cannot + rely on the search paths - CI, for one - names the file it found instead. + """ + if profile is None and os.environ.get("PDF_GRAY_PROFILE"): + profile = Path(os.environ["PDF_GRAY_PROFILE"]) if profile is not None: if not profile.is_file(): raise click.ClickException(f"ICC profile {profile} not found") From 8667f878bcf37af86cfbeea4f62ce7c8223f0e28 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Wed, 16 Sep 2026 17:35:23 +0200 Subject: [PATCH 14/15] run ghostscript from a container in CI Ubuntu ships Ghostscript 10.02, where pdfwrite declares PDFX a boolean, so -dPDFX=4 raises /typecheck in --pdfmark-- and PDF/X-4 cannot be written at all. The workflow installed that one from apt. - build an alpine image with ghostscript and call it through a shim that mounts the workspace and /tmp at the paths the build uses - take the grey output intent profile out of the same image - let build-pdf name the binary with --ghostscript or GHOSTSCRIPT: it ran the conversions with a hardcoded gs, which the runner no longer has - refuse a ghostscript older than 10.03 with the reason, and note one older than the tested 10.08, instead of failing deep inside pdfmark --- .github/workflows/pdf.yml | 37 +++++++++++++++++++---------- tasks/backlog.md | 11 +++++++-- tasks/spec.md | 5 ++++ tests/test_build_pdf_print.py | 13 +++++++++++ tests/test_pdf_normalize.py | 22 ++++++++++++++++++ tools/build_pdf.py | 16 +++++++++++-- tools/pdf_normalize.py | 44 +++++++++++++++++++++++++++++++++++ 7 files changed, 132 insertions(+), 16 deletions(-) diff --git a/.github/workflows/pdf.yml b/.github/workflows/pdf.yml index c3e6228ba..711f0dd80 100644 --- a/.github/workflows/pdf.yml +++ b/.github/workflows/pdf.yml @@ -68,10 +68,10 @@ jobs: - name: update packages run: sudo apt-get update -y - # Ghostscript converts the book block to PDF/X-4 and to greyscale; poppler - # carries the pdffonts, pdfimages and pdftotext the preflight report reads. + # Poppler carries the pdffonts, pdfimages and pdftotext the preflight report + # reads. Ghostscript comes from a container below, not from apt. - name: install packages - run: sudo apt-get install -y libffi-dev libz-dev ghostscript poppler-utils + run: sudo apt-get install -y libffi-dev libz-dev poppler-utils - name: install pandoc run: | @@ -95,16 +95,29 @@ jobs: path: dist/icc key: icc-iso-coated-v2-eci - # Where a distribution puts Ghostscript's own grey profile differs, so the - # greyscale pass is told which file to use instead of guessing. - - name: find the grey profile + # Ubuntu ships Ghostscript 10.02, where `-dPDFX=4` is still a boolean, so + # the 4 raises `/typecheck in --pdfmark--` and PDF/X-4 cannot be written at + # all. A container carries a newer one, behind a shim that runs it at the + # paths the build uses, so the PDFs, the pdfmark prefix and the profiles + # resolve inside the container exactly as they do outside. The grey output + # intent profile comes out of the same image. + - name: ghostscript from a container run: | - profile="$(find /usr/share /usr/lib -name default_gray.icc -print -quit 2>/dev/null || true)" - if [ -z "$profile" ]; then - echo "no default_gray.icc found - install it or drop --gray" >&2 - exit 1 - fi - echo "PDF_GRAY_PROFILE=$profile" >> "$GITHUB_ENV" + printf 'FROM alpine:edge\nRUN apk add --no-cache ghostscript\n' | docker build -q -t pdfx-gs - + W="$GITHUB_WORKSPACE" + sudo tee /usr/local/bin/gs-docker >/dev/null < dist/icc/default_gray.icc + { + echo "GHOSTSCRIPT=/usr/local/bin/gs-docker" + echo "PDF_GRAY_PROFILE=$W/dist/icc/default_gray.icc" + } >> "$GITHUB_ENV" - name: build the screen pdf run: task pdf diff --git a/tasks/backlog.md b/tasks/backlog.md index d9d5a1ae1..55b551c29 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -384,8 +384,15 @@ Done 2026-09-16: `.github/workflows/pdf.yml` builds every edition after a push t `feature/print-on-demand`, and on demand from the Actions tab - the screen PDF from `task pdf`, and the book block, its PDF/X-4 copy in CMYK and the greyscale preview from `task pdf:print -- --normalize --gray`. -- **The runner gets Ghostscript and poppler.** The preflight report shells out to `pdffonts`, `pdfimages` - and `pdftotext`, so without poppler the print build fails rather than the check being skipped. +- **The runner gets poppler; Ghostscript comes from a container.** The preflight shells out to `pdffonts`, + `pdfimages` and `pdftotext`, so poppler is installed with apt. Ghostscript is not: Ubuntu ships 10.02, + where `-dPDFX=4` is a boolean and the 4 raises `/typecheck in --pdfmark--` (spec §7). The workflow builds + an `alpine:edge` image with Ghostscript 10.07 and writes a shim to `/usr/local/bin/gs-docker` that runs + it with the workspace and `/tmp` mounted at the paths they already have, so the PDFs, the pdfmark prefix + and the profiles resolve inside the container as they do outside. `GHOSTSCRIPT` points the build at the + shim, and `default_gray.icc` is copied out of the same image into `dist/icc`, which the greyscale pass + finds through `PDF_GRAY_PROFILE`. Verified end to end against that image: the CMYK copy carries the + FOGRA39 intent with four components, the preview the `sGray` intent with one. - **Four artifacts**, one per edition (`pdf-screen`, `pdf-print`, `pdf-print-x4`, `pdf-print-gray-x4`), kept 30 days and stored uncompressed, since a PDF is compressed already. - **The run summary** lists each edition with its page count, size and download link, so nothing has to be diff --git a/tasks/spec.md b/tasks/spec.md index 563bc259d..65eef7e49 100644 --- a/tasks/spec.md +++ b/tasks/spec.md @@ -345,6 +345,11 @@ Traps found: - `-sOutputICCProfile` with `-dPDFX` crashes the PDF interpreter (`/undefined in --runpdf--`) and leaves a truncated file - supply the profile only through the prefix file, which reads it with `--permit-file-read=`. +- **PDF/X-4 needs Ghostscript 10.03 or newer.** Before that, pdfwrite declares `PDFX` a boolean + (`gs_param_type_bool` in `devices/vector/gdevpdfp.c`), so the 4 of `-dPDFX=4` raises + `/typecheck in --pdfmark--` while the prefix runs. Ubuntu 24.04 ships 10.02 and cannot write PDF/X-4 at + all; its own `PDFX_def.ps` only ever writes `PDF/X-3:2002`. The build says so before it starts, and CI + runs Ghostscript from a container instead (backlog P16). - Ghostscript ships a generic `default_cmyk.icc`, not FOGRA39. The ISO Coated v2 profile comes from the ECI. **Not vendored** (checked 2026-09-15): the ECI only says the profiles may be "freely downloaded", and the profile's own copyright reads "© Heidelberger Druckmaschinen AG. All Rights Reserved". The diff --git a/tests/test_build_pdf_print.py b/tests/test_build_pdf_print.py index d88d7041f..9a452f08d 100644 --- a/tests/test_build_pdf_print.py +++ b/tests/test_build_pdf_print.py @@ -10,6 +10,7 @@ NavEntry, SectionRule, apply_section_rules, + build_pdf, keep_lead_with_heading, load_section_rules, md_to_built_html, @@ -377,3 +378,15 @@ def test_a_short_lead_line_is_kept_with_its_heading(): "Short, under a heading the rule still covers.", ] assert stats["lead lines kept with their heading"] == 2 + + +def test_the_ghostscript_of_the_conversions_can_be_named(monkeypatch): + """CI runs Ghostscript from a container, so the binary must be nameable - by option and by variable.""" + monkeypatch.delenv("GHOSTSCRIPT", raising=False) + context = build_pdf.make_context("build-pdf", ["--edition", "print"]) + assert context.params["ghostscript"] == "gs" + context = build_pdf.make_context("build-pdf", ["--edition", "print", "--ghostscript", "/usr/local/bin/gs-docker"]) + assert context.params["ghostscript"] == "/usr/local/bin/gs-docker" + monkeypatch.setenv("GHOSTSCRIPT", "/usr/local/bin/gs-docker") + context = build_pdf.make_context("build-pdf", ["--edition", "print"]) + assert context.params["ghostscript"] == "/usr/local/bin/gs-docker" diff --git a/tests/test_pdf_normalize.py b/tests/test_pdf_normalize.py index 3cd0cde01..58ce70fde 100644 --- a/tests/test_pdf_normalize.py +++ b/tests/test_pdf_normalize.py @@ -14,6 +14,8 @@ ensure_gray_profile, fetch_profile, ghostscript_command, + ghostscript_is_tested, + ghostscript_writes_pdfx4, gray_command, gray_path, gray_preview, @@ -70,6 +72,26 @@ def test_each_output_is_named_after_what_it_is(): assert gray_path(book).name == "documentation-eccenca-com-26-2-print-gray-x4.pdf" +@pytest.mark.parametrize( + "version, writes_pdfx4, tested", + [ + ("10.08.0", True, True), + ("10.10.0", True, True), + ("11.0.0", True, True), + ("10.07.1", True, False), # writes PDF/X-4, but is not the verified release + ("10.03.0", True, False), + ("10.02.1", False, False), # -dPDFX is a boolean here + ("GPL Ghostscript 10.02.1 (2023-11-01)", False, False), + ("9.55.0", False, False), + ("", True, True), # an unknown version says nothing + ("Page 1\nPage 2", True, True), # nor does a wrapper that answers with something of its own + ], +) +def test_the_ghostscript_version_is_judged_by_what_it_can_do(version, writes_pdfx4, tested): + assert ghostscript_writes_pdfx4(version) is writes_pdfx4 + assert ghostscript_is_tested(version) is tested + + def test_the_grey_profile_can_be_named_by_the_environment(tmp_path, monkeypatch): profile = tmp_path / "gray.icc" profile.write_bytes(b"icc profile bytes") diff --git a/tools/build_pdf.py b/tools/build_pdf.py index a93184c82..ca679ea2b 100644 --- a/tools/build_pdf.py +++ b/tools/build_pdf.py @@ -1771,6 +1771,13 @@ def unpadded_pages(compile_command: list[str], typ_path: Path) -> int: help="Also write a greyscale preview with Ghostscript, to see the black-and-white print on screen? " "Print edition only.", ) +@click.option( + "--ghostscript", + default="gs", + envvar="GHOSTSCRIPT", + show_default=True, + help="Which Ghostscript converts the book block? A wrapper works, for one that runs in a container.", +) def build_pdf( build_version: str, edition_name: str, @@ -1779,6 +1786,7 @@ def build_pdf( typst_binary: str, normalize_output: bool, gray_output: bool, + ghostscript: str, ) -> None: """Build a single PDF of the whole site with pandoc and Typst.""" version = build_version.strip() or "dev" @@ -1912,10 +1920,14 @@ def build_pdf( if print_edition and not print_report(out, run_preflight(out)): raise click.ClickException(f"preflight found errors in {out} - it is written, but not ready for print") if normalize_output: - final = normalize_pdf(out, normalized_path(out), ensure_profile(), title=f"{BOOK_TITLE} - {BOOK_CONTEXT}") + final = normalize_pdf( + out, normalized_path(out), ensure_profile(), ghostscript, title=f"{BOOK_TITLE} - {BOOK_CONTEXT}" + ) print(f"PDF/X-4 written to {final} ({final.stat().st_size // 1024} KB)") if not print_report(final, run_preflight(final, pdfx=True)): raise click.ClickException(f"preflight found errors in {final} - it is written, but not ready for print") if gray_output: - preview = gray_preview(out, gray_path(out), title=f"{BOOK_TITLE} - {BOOK_CONTEXT}") + preview = gray_preview( + out, gray_path(out), ghostscript=ghostscript, title=f"{BOOK_TITLE} - {BOOK_CONTEXT}" + ) print(f"Greyscale preview written to {preview} ({preview.stat().st_size // 1024} KB)") diff --git a/tools/pdf_normalize.py b/tools/pdf_normalize.py index 6455dd6d5..2177a0a9c 100644 --- a/tools/pdf_normalize.py +++ b/tools/pdf_normalize.py @@ -25,8 +25,10 @@ import hashlib import io import os +import re import shutil import subprocess +import sys import tempfile import time import urllib.error @@ -41,6 +43,13 @@ PROFILE_MEMBER = "ECI_Offset_2009/ISOcoated_v2_eci.icc" PROFILE_SHA256 = "128dc02f7246cc3807af0323695379f64151a8f27a587736acc59f8b6ce894b8" PROFILE_CACHE = Path("dist/icc/ISOcoated_v2_eci.icc") +# The recipe of tasks/spec.md, §7 was verified with this release. +TESTED_GHOSTSCRIPT = "10.08" +# PDF/X-4 needs `-dPDFX=4`, and pdfwrite took that parameter as an integer only +# from 10.03 on. Before that it is a boolean (`gs_param_type_bool` in +# devices/vector/gdevpdfp.c), so the 4 raises `/typecheck in --pdfmark--` - +# Ubuntu 24.04 ships 10.02 and cannot write PDF/X-4 at all. +PDFX4_GHOSTSCRIPT = "10.03" FOGRA39 = "FOGRA39" FOGRA39_INFO = "ISO Coated v2 (ECI)" COLOR_REGISTRY = "http://www.color.org" @@ -207,10 +216,45 @@ def gray_path(pdf: Path) -> Path: return pdf.with_name(f"{pdf.stem}-gray-x4.pdf") +def ghostscript_at_least(version: str, wanted: str) -> bool: + """Whether a Ghostscript version string names `wanted` or a newer release. + + Only a dotted `major.minor` counts as a version; anything else - a wrapper + that answers `--version` with something of its own - passes unjudged. + """ + found = re.search(r"(\d+)\.(\d+)", version) + if found is None: + return True + return tuple(int(number) for number in found.groups()) >= tuple(int(n) for n in wanted.split(".")) + + +def ghostscript_is_tested(version: str) -> bool: + """Whether a Ghostscript version is the release series the recipe was verified with, or a newer one.""" + return ghostscript_at_least(version, TESTED_GHOSTSCRIPT) + + +def ghostscript_writes_pdfx4(version: str) -> bool: + """Whether a Ghostscript version can write PDF/X-4 at all - `-dPDFX=4` is an integer only from 10.03 on.""" + return ghostscript_at_least(version, PDFX4_GHOSTSCRIPT) + + def ghostscript_binary(ghostscript: str) -> str: + """The Ghostscript to run, with a word about its version - an old one fails deep inside pdfmark.""" binary = shutil.which(ghostscript) if binary is None: raise click.ClickException(f"{ghostscript} not found - install Ghostscript or name the binary with --ghostscript") + version = subprocess.run([binary, "--version"], capture_output=True, text=True, check=False).stdout.strip() + if version and not ghostscript_writes_pdfx4(version): + raise click.ClickException( + f"Ghostscript {version} cannot write PDF/X-4: `-dPDFX=4` is a boolean before {PDFX4_GHOSTSCRIPT} and " + "the 4 raises `/typecheck in --pdfmark--`. Name a newer one with --ghostscript or GHOSTSCRIPT " + f"(the recipe is verified with {TESTED_GHOSTSCRIPT})." + ) + if version and not ghostscript_is_tested(version): + print( + f"NOTE: Ghostscript {version} is older than the {TESTED_GHOSTSCRIPT} the recipe was verified with.", + file=sys.stderr, + ) return binary From ee9d6d817e85dd5bde61ea98e697568208826590 Mon Sep 17 00:00:00 2001 From: Rene Pietzsch Date: Thu, 17 Sep 2026 09:52:37 +0200 Subject: [PATCH 15/15] write the pdf table into the run summary The step passed but the summary stayed empty: its rows went through a shell function whose redirect never reached GITHUB_STEP_SUMMARY, and a silent non-write survives bash -e. - build the table in one block and tee it, so the editions and their links show up in the step log as well - print the size of the summary file afterwards - fall back to a "missing" row instead of a blank one - pick the screen edition by excluding -print: the glob for it also matched the greyscale copy, whose name ends in a digit too --- .github/workflows/pdf.yml | 43 ++++++++++++++++++++++----------------- tasks/backlog.md | 5 ++++- 2 files changed, 28 insertions(+), 20 deletions(-) diff --git a/.github/workflows/pdf.yml b/.github/workflows/pdf.yml index 711f0dd80..65d23083c 100644 --- a/.github/workflows/pdf.yml +++ b/.github/workflows/pdf.yml @@ -168,33 +168,38 @@ jobs: retention-days: 30 compression-level: 0 + # The table is built in one go and printed to the log as well, so the + # editions and their links are readable even if the summary ever stays + # empty - which is how the first run of this step ended. - name: list the pdfs in the run summary run: | - row() { - file="$1" - edition="$2" - url="$3" - pages="$(pdfinfo "$file" | awk '/^Pages/ {print $2}')" - size="$(du -h "$file" | cut -f1)" - printf '| %s | %s | %s | [%s](%s) |\n' \ - "$edition" "$pages" "$size" "$(basename "$file")" "$url" >> "$GITHUB_STEP_SUMMARY" + # $1 is a glob; $4, when given, drops what the glob also matches - the + # screen edition is the one file whose name carries no `-print`. + describe() { + file="$(ls $1 2>/dev/null | grep -v -- "${4:-^$}" | head -1)" + if [ -z "$file" ]; then + printf '| %s | - | - | missing |\n' "$2" + return + fi + printf '| %s | %s | %s | [%s](%s) |\n' "$2" \ + "$(pdfinfo "$file" 2>/dev/null | awk '/^Pages/ {print $2}')" \ + "$(du -h "$file" | cut -f1)" "$(basename "$file")" "$3" } { echo "## PDF editions of ${GITHUB_REF_NAME}" echo echo "| Edition | Pages | Size | Download |" echo "| :------ | ----: | ---: | :------- |" - } >> "$GITHUB_STEP_SUMMARY" - row "$(ls dist/documentation-eccenca-com-*.pdf | grep -v -- '-print' | head -1)" \ - "Screen" "${{ steps.screen.outputs.artifact-url }}" - row "$(ls dist/documentation-eccenca-com-*-print.pdf | head -1)" \ - "Print, book block" "${{ steps.print.outputs.artifact-url }}" - row "$(ls dist/documentation-eccenca-com-*-print-x4.pdf | head -1)" \ - "Print, PDF/X-4 in CMYK" "${{ steps.pdfx.outputs.artifact-url }}" - row "$(ls dist/documentation-eccenca-com-*-print-gray-x4.pdf | head -1)" \ - "Print, greyscale preview" "${{ steps.gray.outputs.artifact-url }}" - { + describe "dist/documentation-eccenca-com-*.pdf" \ + "Screen" "${{ steps.screen.outputs.artifact-url }}" "-print" + describe "dist/documentation-eccenca-com-*-print.pdf" \ + "Print, book block" "${{ steps.print.outputs.artifact-url }}" + describe "dist/documentation-eccenca-com-*-print-x4.pdf" \ + "Print, PDF/X-4 in CMYK" "${{ steps.pdfx.outputs.artifact-url }}" + describe "dist/documentation-eccenca-com-*-print-gray-x4.pdf" \ + "Print, greyscale preview" "${{ steps.gray.outputs.artifact-url }}" echo echo "The print editions carry the preflight report of \`dec-tool pdf-preflight\`;" echo "a failed check fails this job." - } >> "$GITHUB_STEP_SUMMARY" + } | tee -a "$GITHUB_STEP_SUMMARY" + echo "summary file: $GITHUB_STEP_SUMMARY, $(wc -c < "$GITHUB_STEP_SUMMARY") bytes" diff --git a/tasks/backlog.md b/tasks/backlog.md index 55b551c29..2152a6576 100644 --- a/tasks/backlog.md +++ b/tasks/backlog.md @@ -396,7 +396,10 @@ book block, its PDF/X-4 copy in CMYK and the greyscale preview from `task pdf:pr - **Four artifacts**, one per edition (`pdf-screen`, `pdf-print`, `pdf-print-x4`, `pdf-print-gray-x4`), kept 30 days and stored uncompressed, since a PDF is compressed already. - **The run summary** lists each edition with its page count, size and download link, so nothing has to be - dug out of the artifact section (user decision 2026-09-16: summary links, no release page). + dug out of the artifact section (user decision 2026-09-16: summary links, no release page). The first + run wrote nothing there, although the step passed, so the table is now built in one block, printed to + the log with `tee` as well, and followed by the byte count of the summary file - a silent non-write + cannot pass for success again. **Open:** the next run has to show the table in both places. - **The output intent profile** is cached in `dist/icc` between runs; its licence keeps it out of the repository, and CI would otherwise fetch it from the ECI every time. - **The grey profile** is located with `find` and passed as `PDF_GRAY_PROFILE`, because distributions put