out_opentelemetry: preserve failed log batch results

[grolegor]

Fixes #12384.

Problem and fix

One input chunk can produce several OTLP log requests. If an earlier request
returns HTTP 429 and a later request succeeds, the later FLB_OK overwrites
FLB_RETRY. Fluent Bit releases the chunk without retrying the rejected logs.

Stop the batching loop on the first non-OK result, after resetting the freed
batch. The existing cleanup path then returns the failure to the engine.

With two records and batch_size: 1:

Before: A -> 429, B -> 200. No retry; A is lost.
After:  A -> 429. Retry: A -> 200, B -> 200. Both arrive.

This retains whole-chunk retry: if A was accepted before B failed, A can be
duplicated on retry. It does not add exactly-once delivery, change HTTP status
classification, change retry limits, or modify the metrics/traces exporters.

A non-retryable result, such as HTTP 400, is also preserved: the engine drops
the chunk instead of reporting success. Later batches are not sent after that
failure. The tests cover both first- and last-batch HTTP 400 responses.

Tests

The 15 new integration cases use either one JSON array sent to the HTTP input
or one native OTLP request containing separate resource groups. They decode
received OTLP payloads and check per-event delivery, complete batches, retries,
permanent failures, and creation/destruction of one task. The standalone
three-file reproduction remains available in the issue.

Local verification used Linux/arm64 in Docker. The patch was built on current
master (f78472443d32691c319c424c75d9ea2343dbc8f7) and on tag v5.1.1.

All 40 OpenTelemetry integration cases pass on patched master, both normally
and with strict Valgrind 3.18.1. All 40 memory reports show zero errors and no
remaining heap allocations. On the unpatched binary, seven of the 15 new cases
fail as expected and eight controls pass. A separate real integration run with
injected metrics ConnectionError and Timeout also passes.

Both DCO-signed commits pass the repository's commit-prefix checker over the
full PR range against master.

Commands run inside the development container, from tests/integration:

.venv/bin/python -m pytest scenarios/out_opentelemetry -q -o log_cli=false --tb=short
VALGRIND=1 VALGRIND_STRICT=1 .venv/bin/python -m pytest scenarios/out_opentelemetry -q -o log_cli=false --tb=short

From the repository root:

ctest --test-dir build -R '^flb-it-opentelemetry$' --output-on-failure

CTest result: 1/1 target passed. Representative Valgrind output (the same clean
summary was verified for all 40 tested processes):

in use at exit: 0 bytes in 0 blocks
All heap blocks were freed -- no leaks are possible
ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Example test configuration

The integration harness provides the ports, TEST_LOG_BATCH_SIZE, and
TEST_LOG_INPUT (http or opentelemetry). Default body keys support both inputs.

service:
  flush: 1
  grace: 1
  log_level: debug
  http_server: on
  http_port: ${FLUENT_BIT_HTTP_MONITORING_PORT}
  scheduler.base: 1
  scheduler.cap: 2

pipeline:
  inputs:
    - name: ${TEST_LOG_INPUT}
      port: ${FLUENT_BIT_TEST_LISTENER_PORT}

  outputs:
    - name: opentelemetry
      match: "*"
      host: 127.0.0.1
      port: ${TEST_SUITE_HTTP_PORT}
      grpc: off
      http2: off
      logs_uri: /v1/logs
      batch_size: ${TEST_LOG_BATCH_SIZE}
      retry_limit: no_limits

Fixed 5.1.1 reproduction output

Request 1: HTTP 429, 1 records, event-000000..event-000000
[debug] [retry] new retry created for task_id=0 attempts=1
Request 2: HTTP 200, 1 records, event-000000..event-000000
Request 3: HTTP 200, 1 records, event-000001..event-000001
[debug] [task] destroy task=0xf0cf7c0613d0 (task_id=0)
Engine successfully processed: 2
Receiver accepted: 2; missing: 0; duplicates: 0
Responses: [429, 200, 200]

The inverse order delivers both records with one duplicate. Both failure orders
also deliver all 2000 records with batch_size: 1000; an initial successful
batch produces 1000 duplicates on whole-chunk retry. Each case has one task.

Submission checklist

Testing

• Example configuration file for the change (above).
• Debug log output from testing the change (above).
• Valgrind output showing no leaks or memory corruption (above).
• [N/A] Run local packaging test: no packaging or build-system changes.
• [N/A] Set ok-package-test label: no packaging or build-system changes.

Documentation

• [N/A] Documentation required for this feature: no options added or changed.

Backporting

• Backport to latest stable release (not included in this PR).

Developed with AI assistance; verification used real local Fluent Bit processes
and a test receiver, not mocked delivery results.

Fluent Bit is licensed under Apache 2.0; by submitting this pull request I
understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• Bug Fixes

  • Improved OpenTelemetry log delivery error handling. When a batch flush fails, processing now stops promptly and preserves the failure result instead of allowing later batches to mask it.
  • Improved reliability for batched HTTP log delivery, including retry scenarios and failed requests.

• Tests

  • Added integration coverage for batching, retries, failure responses, dropped records, completion metrics, and request lifecycle behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Team

Run ID: 4d7cf912-be9c-4c8a-9869-a79a42084f3d

📥 Commits

Reviewing files that changed from the base of the PR and between f784724 and a2f6c43.

📒 Files selected for processing (3)

• plugins/out_opentelemetry/opentelemetry_logs.c
• tests/integration/scenarios/out_opentelemetry/config/out_otel_http_logs_batch_retry.yaml
• tests/integration/scenarios/out_opentelemetry/tests/test_out_opentelemetry_001.py

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.

---

📝 Walkthrough

Walkthrough

The OpenTelemetry log processor now stops decoding after a failed batch flush. A new HTTP integration scenario tests batching, retryable and permanent failures, record handling, metrics, emitted batches, and task lifecycle behavior.

Changes

OpenTelemetry log batch retry

Layer / File(s)	Summary
Preserve batch flush status plugins/out_opentelemetry/opentelemetry_logs.c	The log processing loop stops when logs_flush_to_otel returns a non-FLB_OK status.
Validate HTTP batch retry behavior tests/integration/scenarios/out_opentelemetry/config/out_otel_http_logs_batch_retry.yaml, tests/integration/scenarios/out_opentelemetry/tests/test_out_opentelemetry_001.py	The integration scenario configures OpenTelemetry HTTP batching with unlimited retries. The parametrized test checks retry and permanent-failure counters, record handling, request batches, completion metrics, and task lifecycle logging.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to a2f6c

This prevents failed OpenTelemetry log batches from being reported as successful and is ready to merge with the supplied retry, failure, lifecycle, and memory-safety validation.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 2 files. (1 skipped: 1 …	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The implementation preserves the first non-OK flush result and stops further batching. The integration test covers retryable HTTP 429 failures, non-retryable failures, batch ordering, partial batches,…
Out of Scope Changes check	✅ Passed	All changes support issue #12384. The source change fixes failure-result preservation, and the added configuration and integration coverage validate the affected batching and retry behavior. No unrela…
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: preserving failed OpenTelemetry log batch results.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 5 functions across 2 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[grolegor]

Hi @cosmo0920, could you please rerun these two CI jobs when you have a chance?

• Linux GCC, SIMD off: downloading 106 MB of apt packages took 56 minutes. The job then hit its 60-minute timeout during compilation, before tests ran. This also caused the aggregate “Unit tests (matrix)” check to fail.
• Windows x86: test_non_blocking_and_blocking_timeout measured 2484 ms against an expected 2100 ±30 ms. This unchanged test does not exercise the OpenTelemetry output; a rerun would help confirm whether this was a transient timing failure.

Thank you!