Skip to content

RemoteFortressReader: send creature/caste names and descriptions as UTF-8 in GetPartialCreatureRaws - #6021

Open
dwmuller wants to merge 1 commit into
DFHack:developfrom
dwmuller:fix/rfr-creature-raws-utf8
Open

dwmuller wants to merge 1 commit into
DFHack:developfrom
dwmuller:fix/rfr-creature-raws-utf8

Conversation

@dwmuller

Copy link
Copy Markdown

Fixes #6019

GetCreatureRaws / GetPartialCreatureRaws copied several DF strings (which are CP437) into protobuf string fields without conversion, so clients that validate UTF-8 (e.g. prost/Rust) reject the entire reply whenever a name contains a non-ASCII character such as å (0x86 in CP437, e.g. "Råsh").

This wraps the affected calls in DF2UTF, as is already done for material names, tissue names, etc.:

  • CreatureRaw.name
  • CasteRaw.caste_name
  • CasteRaw.baby_name / child_name
  • CreatureRaw.general_baby_name / general_child_name
  • CasteRaw.description

Identifier-like tokens (creature_id, caste_id, body part tokens, etc.) are intentionally left alone, since they should be ASCII.

A changelog entry is included under Future → Fixes.

Testing

  • Built RemoteFortressReader (RelWithDebInfo, MSVC 2022) from this branch; it matches the 53.16-r3 tag apart from the scripts submodule pointer.
  • Installed the patched plugin into a Steam DF 53.16 / DFHack 53.16-r3 install and ran live tests against a fortress that previously triggered the failure. GetCreatureRaws now decodes successfully and a raw wire scan finds no non-UTF-8 strings. The tests are in https://github.com/dwmuller/DFCompanion (tests/live_game_tests.rs).

Compatibility

  • I am confident that this will not cause problems for clients that may be doing the conversion themselves. UTF-8 characters cannot be mistaken for code page characters that need conversion, so a client-side conversion will have no effect.
  • This fixes a violation of the protobuf2 standard in that the structures defined for protobuf by RemoteFortressReader specify these as string fields, and protobuf2 strings must be either UTF-8 or ASCII.

Prepared with GitHub Copilot under the direction of @dwmuller, who reviewed, built and tested the change against a live game.

…o UTF-8 in GetPartialCreatureRaws

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

RemoteFortressReader: GetCreatureRaws sends CP437 (not UTF-8) strings, which breaks protobuf decoding

1 participant