Skip to content

T-19629 Keep sending the batch when a log argument can't be serialized - #27

Draft
PetrHeinz wants to merge 3 commits into
mainfrom
claude/t-19629-cyclic-args
Draft

T-19629 Keep sending the batch when a log argument can't be serialized#27
PetrHeinz wants to merge 3 commits into
mainfrom
claude/t-19629-cyclic-args

Conversation

@PetrHeinz

@PetrHeinz PetrHeinz commented Jul 23, 2026

Copy link
Copy Markdown
Member

Fixes #26.

buildPostData() puts the raw SLF4J argument array into the payload and batchToJson() serializes the whole batch with a plain ObjectMapper, which has no cycle detection. One log line carrying a cyclic object graph — a JDBC Connection, which HikariCP logs on every pooled connection creation — makes serialization of the entire batch throw, and flushLogs() then sets retries = maxRetries and drops all of it.

The fix is a BestEffortSerialization Jackson module registered on the appender's mapper, so there is exactly one serialization pass and one way of representing data:

  • A BeanSerializerModifier wraps the bean, map, collection and array serializers with a cycle guard that keeps an identity set of the objects currently being serialized — the same approach as logtail-python's frame.py. A reference looping back into the graph is replaced with "<omitted circular reference>" and everything around it stays structured JSON: a cyclic Parent argument arrives as {"Child":{"Parent":"<omitted circular reference>"}}. Only true ancestors count, so the same object referenced twice in a DAG still serializes fully both times.
  • SLF4J arguments — the only place arbitrary objects enter the payload — are additionally guarded: each argument serializes once into a TokenBuffer that is then replayed into the output stream. If it fails for a reason other than a cycle (typically a getter that throws), the buffer is discarded and the whole argument is sent as "<omitted unserializable com.example.Foo>".

toString() is never called on logged objects, nothing is serialized twice, and happy-path output is byte-identical to before. The one trade-off: a bean with one good getter and one throwing getter is omitted as a whole argument — there is no clean way to back out mid-object in a streaming write, and an explicit marker naming the class beats a half-object.

The first commit is the failing reproduction on its own, so the diff shows the bug reproducing before it shows it fixed. The second commit is an earlier sanitize-and-retry attempt kept in history; the final commit replaces it with the module.

Worth noting for anyone reproducing this: on the jackson-databind 2.13.5 this repo pins, the cycle surfaces as Infinite recursion (StackOverflowError), while on the reporter's 2.21.4 the same cycle trips the StreamWriteConstraints nesting-depth guard instead (Document nesting depth (1001) exceeds the maximum allowed, a guard that only exists from Jackson 2.15). With the cycle guard neither is ever thrown — the cycle is cut before the serializer can recurse.

The 9 integration tests that need BETTER_STACK_SOURCE_TOKEN fail identically on main when run without the secret — unrelated to this change.

🤖 Generated with Claude Code

PetrHeinz and others added 2 commits July 23, 2026 10:13
A log argument with a cyclic object graph makes batchToJson() throw, which
makes flushLogs() give up on the entire batch. The test asserts that the
surrounding log lines still get serialized; it currently fails with
JsonMappingException: Infinite recursion (StackOverflowError).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
batchToJson() now falls back to sanitizing the batch when the ObjectMapper
throws: log lines that serialize fine are kept untouched, and only the values
Jackson chokes on are replaced with their string representation. A cyclic
object graph on one log line - HikariCP logs a pooled JDBC connection on every
connection creation - no longer costs the whole batch.

The fallback only runs after a failed serialization, so the happy path is
unchanged. StackOverflowError is caught alongside Exception because Jackson
only wraps it into JsonMappingException for bean serializers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@PetrHeinz PetrHeinz changed the title T-19629 Reproduction test: cyclic log argument drops the entire batch T-19629 Keep sending the batch when a log argument can't be serialized Jul 23, 2026
Replaces the sanitize-and-retry fallback: there is now exactly one
serialization per batch and one way of representing data. A new
BestEffortSerialization Jackson module detects circular references
during serialization (identity set of ancestor objects, mirroring
logtail-python's frame.py) and replaces the reference back into the
graph with "<omitted circular reference>" while the rest of the object
stays structured JSON. SLF4J arguments - the only place arbitrary
objects enter the payload - are additionally guarded: an argument whose
serialization fails for any other reason (typically a getter that
throws) is buffered through a TokenBuffer and replaced as a whole with
"<omitted unserializable <class>>". toString() is never called on
logged objects and nothing is ever serialized twice.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LogtailAppender crashes on any log argument with a cyclic object graph (e.g. a JDBC Connection), dropping the entire batch

1 participant