T-19629 Keep sending the batch when a log argument can't be serialized - #27
Draft
PetrHeinz wants to merge 3 commits into
Draft
T-19629 Keep sending the batch when a log argument can't be serialized#27PetrHeinz wants to merge 3 commits into
PetrHeinz wants to merge 3 commits into
Conversation
A log argument with a cyclic object graph makes batchToJson() throw, which makes flushLogs() give up on the entire batch. The test asserts that the surrounding log lines still get serialized; it currently fails with JsonMappingException: Infinite recursion (StackOverflowError). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
batchToJson() now falls back to sanitizing the batch when the ObjectMapper throws: log lines that serialize fine are kept untouched, and only the values Jackson chokes on are replaced with their string representation. A cyclic object graph on one log line - HikariCP logs a pooled JDBC connection on every connection creation - no longer costs the whole batch. The fallback only runs after a failed serialization, so the happy path is unchanged. StackOverflowError is caught alongside Exception because Jackson only wraps it into JsonMappingException for bean serializers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the sanitize-and-retry fallback: there is now exactly one serialization per batch and one way of representing data. A new BestEffortSerialization Jackson module detects circular references during serialization (identity set of ancestor objects, mirroring logtail-python's frame.py) and replaces the reference back into the graph with "<omitted circular reference>" while the rest of the object stays structured JSON. SLF4J arguments - the only place arbitrary objects enter the payload - are additionally guarded: an argument whose serialization fails for any other reason (typically a getter that throws) is buffered through a TokenBuffer and replaced as a whole with "<omitted unserializable <class>>". toString() is never called on logged objects and nothing is ever serialized twice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #26.
buildPostData()puts the raw SLF4J argument array into the payload andbatchToJson()serializes the whole batch with a plainObjectMapper, which has no cycle detection. One log line carrying a cyclic object graph — a JDBCConnection, which HikariCP logs on every pooled connection creation — makes serialization of the entire batch throw, andflushLogs()then setsretries = maxRetriesand drops all of it.The fix is a
BestEffortSerializationJackson module registered on the appender's mapper, so there is exactly one serialization pass and one way of representing data:BeanSerializerModifierwraps the bean, map, collection and array serializers with a cycle guard that keeps an identity set of the objects currently being serialized — the same approach as logtail-python'sframe.py. A reference looping back into the graph is replaced with"<omitted circular reference>"and everything around it stays structured JSON: a cyclicParentargument arrives as{"Child":{"Parent":"<omitted circular reference>"}}. Only true ancestors count, so the same object referenced twice in a DAG still serializes fully both times.TokenBufferthat is then replayed into the output stream. If it fails for a reason other than a cycle (typically a getter that throws), the buffer is discarded and the whole argument is sent as"<omitted unserializable com.example.Foo>".toString()is never called on logged objects, nothing is serialized twice, and happy-path output is byte-identical to before. The one trade-off: a bean with one good getter and one throwing getter is omitted as a whole argument — there is no clean way to back out mid-object in a streaming write, and an explicit marker naming the class beats a half-object.The first commit is the failing reproduction on its own, so the diff shows the bug reproducing before it shows it fixed. The second commit is an earlier sanitize-and-retry attempt kept in history; the final commit replaces it with the module.
Worth noting for anyone reproducing this: on the jackson-databind 2.13.5 this repo pins, the cycle surfaces as
Infinite recursion (StackOverflowError), while on the reporter's 2.21.4 the same cycle trips theStreamWriteConstraintsnesting-depth guard instead (Document nesting depth (1001) exceeds the maximum allowed, a guard that only exists from Jackson 2.15). With the cycle guard neither is ever thrown — the cycle is cut before the serializer can recurse.The 9 integration tests that need
BETTER_STACK_SOURCE_TOKENfail identically onmainwhen run without the secret — unrelated to this change.🤖 Generated with Claude Code