allure-karate: afterScenario hook failures can leave scenarios reported as passed

[andrcuns]

What happened?

A Karate scenario with a passing body and a failing configure afterScenario hook can be reported by Allure as passed, without status details or a reported teardown failure.

The hook does execute, and Karate records its exception. The Allure adapter finalizes the scenario before that exception is added to Karate's result. This is a reporting correctness problem: scenario cleanup can fail without affecting the reported scenario outcome.

Steps to reproduce

The repository already contains this reproduction feature:

Feature: Scenario and feature cleanup failures

Background:
  * configure afterScenario = function(){ throw new Error('scenario cleanup failed') }
  * configure afterFeature = function(){ throw new Error('feature cleanup failed') }

Scenario: Scenario with both cleanup hooks
  * match 1 == 1

Run the existing test through Allure agent mode:

npx --yes allure@3.17.0 agent --report off \
  --results-dir allure-karate/build/allure-results \
  --expect-tests 1 \
  --expect-prefix io.qameta.allure.karate. \
  -- ./gradlew --no-daemon :allure-karate:cleanTest :allure-karate:test \
  --tests io.qameta.allure.karate.AllureKarateGlobalErrorsTest.shouldReportOnlyFeatureHookFailuresAsGlobals

Open the generated agent index.md, then inspect the test's attached inner Karate TestResult and globals payloads.

Important: the existing outer JUnit test checks global-error scoping and therefore passes. It does not assert that the inner scenario's teardown failure is reported correctly.

Actual behavior

The captured Allure scenario result has these relevant fields:

{
  "name": "Scenario with both cleanup hooks",
  "status": "passed"
}

There are no status details or failed teardown steps for scenario cleanup failed. The configuration steps and the ordinary match step are reported as passed.

The separate afterFeature exception is correctly present in globals as feature cleanup failed. It is included in the reproducer to demonstrate the distinction between scenario-scoped and feature-scoped cleanup failures.

Expected behavior

• A failing afterScenario hook must be associated with its own scenario and produce an appropriate non-passing outcome, with its exception details available in the report.
• If the scenario body also fails, the primary failure and the additional cleanup failure should both remain visible.
• Per-scenario hook failures must not be converted into framework-wide global errors.
• Shared afterFeature and afterScenarioOutline failures must remain separate from the scenario outcome.

Investigation

In Karate 2.1.2, the relevant order in ScenarioRuntime.call() is:

1. Emit SCENARIO_EXIT.
2. Enter the finally block.
3. Invoke afterScenario and append its hook result, including any exception.

In AllureKarate.afterScenario(), the adapter removes the scenario context, reads the outcome, and calls stopTest and writeTest while handling step 1. The later hook failure therefore arrives after the Allure result has been finalized.

The feature-completion global-error handling intentionally excludes per-scenario hook results. That is the correct scope distinction, but it does not repair the premature scenario finalization.

A fix needs a completion point after scenario teardown, while still separating shared-hook errors from scenario errors. Merely copying the final Karate scenario status at feature completion would also be unsafe, because Karate appends shared-hook failures to scenario results too.

Regression coverage should include a passing body with failing teardown, a failing body with failing teardown, report suppression (including suppressed callees), and parallel/nested execution. Any hook logs or attachments should also be checked when changing the completion boundary.

What Allure Integration are you using?

allure-karate

What version of Allure Integration you are using?

3.1-SNAPSHOT, development revision 813f4b90b5e8a06865473af8bea5721900e1bfd6, with io.karatelabs:karate-core:2.1.2. Observed using JDK 25. This report does not claim that every released adapter version is affected.

What version of Allure Report you are using?

3.17.0. The incorrect outcome is already present in the captured Allure result JSON, before report rendering.

Duplicate check

Searched open and closed issues for Karate, afterScenario, SCENARIO_EXIT, and hook/status symptoms. No matching report of this lifecycle-ordering problem was found.

Code of Conduct

• I agree to follow this project's Code of Conduct