turtlebot3: stop the debounce profile from hiding the goal-status faults - #74
Merged
Conversation
The debounce profile sets confirmation_threshold -3, which suits the no-progress check: it repeats every five seconds while the robot is stuck, so three reports arrive and the profile filters roughly fifteen seconds of it. That is the contrast with the storm profile. The other two faults do not repeat. NAVIGATION_GOAL_ABORTED and NAVIGATION_GOAL_CANCELED are raised from a status change, guarded so they fire once, and cleared by one PASSED when a later goal succeeds. Under -3 the counter reaches -1 and stops, so both stayed PREFAILED for good and the default CONFIRMED-only fault list never showed them. Measured against a running fault_manager: one FAILED leaves PREFAILED at -1, silence changes nothing, and the PASSED only moves the counter to 0. They now report under their own source_id and take their thresholds from a per-source file, -1 to confirm on the single event and 0 to heal on the single clear. The same measurement with that file in place gives CONFIRMED on the raise and HEALED on the recovery, while the no-progress fault still needs its three reports. healing_threshold was 3 with a comment saying three PASSED events heal. Healing costs healing_threshold minus the counter at recovery, so from -3 it would have taken six, and the detector sends one. It is 0. The storm profile set confirmation_threshold 0, which the fault manager rejects and replaces with -1 while logging a warning. It now says -1.
Per-topic snapshots are written on the fault manager capture thread pool. They land a bit after the fault reports CONFIRMED, and CONFIRMED is the status the previous step waits for. The check read the fault detail once at that moment, so it sometimes saw an empty snapshot list while the capture was still running. In one run the check ran 124 ms before the fault manager logged "Captured 3/3 snapshots" for the same fault code. The rosbag check below already polls for the same reason. This makes the snapshot check poll too, with a 30s budget.
mfaferek93
reviewed
Sep 7, 2026
mfaferek93
reviewed
Sep 7, 2026
The per-source threshold key is matched as an anchored prefix of the reported source_id, and the detector reports its fully qualified node name. The launch file puts it in the "bridge" namespace, so the key needs /bridge in front of it; without that the resolver falls back to the global -3 and the goal-status faults stay PREFAILED, where the default fault list never shows them. Healing a confirmed fault costs healing_threshold minus the counter, and the counter is clamped at the confirmation threshold, so a confirmed fault needs a burst of PASSED events rather than one. The detector sent exactly one and then stopped, so NAVIGATION_NO_PROGRESS and LOCALIZATION_UNCERTAINTY could confirm and never heal. A recovery now answers with three, throttled on the same window as the FAILED side. Add a smoke test for the debounce profile and a CI job that runs it, plus the compose override entry for the CI service that job starts. Without that entry the demo comes up on the default profile while the test believes it is debouncing.
The burst was sent from the odometry callback, so it only advanced while the robot kept reporting movement. A recovery that ended before the burst was spent, because the goal finished or the robot stopped, left the remaining PASSED events owed and the fault confirmed for good. The periodic check now sends them whenever the no-progress condition does not hold, which also covers the case where no goal is active and the fault cannot apply. Pin the configured -3 in the smoke test. The old sequence passed against -2 as well, because nothing was asserted between the second and third event in either direction. The test now asserts the intermediate states instead of only refuting one forbidden value, which also passed when the fault moved to some other unexpected status or left the list. Add assertions on the confirmed-only listing, and correct the comment claiming a PREFAILED fault is missing from the default fault list. The default filter includes PREFAILED alongside CONFIRMED, so such a fault is listed; what it never does is count as confirmed.
… owed A successful goal reported PASSED for both goal-status codes whether or not either had ever been raised, so every successful goal cost two service calls for the life of the process. Under a profile with healing disabled a confirmed fault stays confirmed, so each of those also published a further update about a fault that was already over. The clears now go only to codes that still owe PASSED events. The healing burst is counted down when a PASSED is sent, not when it lands, so a report that never reached the fault manager spent part of the budget and left the counter short with nothing left to send. An undelivered or rejected PASSED now puts its share back. Add a static check that the debounce override covers every service that can run the demo, and run it in the debounce job. It also covers the cpu and nvidia services, which CI never starts. Say in the per-source threshold file that its healing fields repeat the global values on purpose, so that source keeps its behaviour if the global profile is retuned.
mfaferek93
approved these changes
Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #73.
The debounce profile's
confirmation_threshold: -3suits the no-progress check, which repeatsevery five seconds while the robot is stuck. It does not suit the other two faults:
NAVIGATION_GOAL_ABORTEDandNAVIGATION_GOAL_CANCELEDcome from a goal status change, fire once,and are cleared by one PASSED on a later success. Under -3 the counter reached -1 and stopped, so
both stayed PREFAILED. They were still listed, because the default fault filter includes PREFAILED
alongside CONFIRMED, but they never counted as confirmed:
confirmedDTCstayed 0, a confirmed-onlyquery did not return them, and nothing that keys off confirmation ran.
They now report under their own
source_idand take their thresholds from a per-source file: -1 toconfirm on the single event, 0 to heal on the single clear. The no-progress fault keeps the global
-3, so the storm-versus-filtered contrast is unchanged.
The key in that file carries the node namespace. The resolver matches it as an anchored prefix of
the reported
source_id, and the detector reports its fully qualified node name, which is/bridge/anomaly_detectorbecause the launch file puts the node in thebridgenamespace. A keywithout the namespace never matches, and the fallback to the global thresholds is silent.
Two smaller fixes on the same files.
healing_thresholdwas 3 under a comment saying three PASSEDevents heal. A confirmed fault sits at the confirmation threshold, because the counter is clamped to
the band, so healing costs
healing_thresholdminus -3: three events at 0, six at 3. The detectorsent one PASSED per recovery and then stopped, so
NAVIGATION_NO_PROGRESSandLOCALIZATION_UNCERTAINTYcould confirm and never heal. A recovery now sends three. Those three aresent from the periodic check rather than from the odometry callback, so a recovery that ends before
the burst is spent, because the goal finishes or the robot stops, does not leave the fault confirmed
for good. And the storm profile set
confirmation_threshold: 0, which the fault manager rejects andreplaces with -1.
Two more on the reporting side. A successful goal cleared both goal-status codes whether or not
either had ever been raised, which cost two service calls per successful goal for the life of the
process and, under a profile with healing disabled, published a further update about a fault that
was already over; the clears now go only to codes that still owe PASSED events. And the burst is
counted down when a PASSED is sent rather than when it lands, so a report that never reached the
fault manager used to spend part of the budget and leave the counter short; an undelivered or
rejected PASSED now puts its share back.
Testing
New
tests/smoke_test_debounce.shdrives the debounce profile through the gateway API and checksthe transitions.
docker-compose.debounce.ymlgained an entry forturtlebot3-demo-ci, withoutwhich that service starts on the default profile while the test believes it is debouncing. A new
build-and-test-turtlebot-debouncejob runs the test.tests/check_debounce_overlay.shguards that last part. It is a static check on the merged composefile, so it also covers the cpu and nvidia services that CI never starts. Removing the override
entry for any of the three fails it.
Measured against a running demo, 23 passed and 0 failed:
The intermediate rows are what pin the configured -3. Without them the same sequence also passes
against -2.
The test was then run against three mutations, to check it fails when the bug comes back:
On a separate run the detector itself raised
NAVIGATION_GOAL_ABORTEDfrom/bridge/anomaly_detector/goal_statuson a real goal abort, and the fault reached CONFIRMED withoccurrence_count: 1. That covers the whole chain: detector,source_id, threshold key, resolver.One thing is not proven end to end: that the detector sends three PASSED on a recovery. The smoke
test reports the events itself, so it pins the fault manager side of that. Driving the detector
would need the robot to get stuck and then move again, and navigation does not come up in the
headless profile at all, which is tracked separately in issue 75.
Also in this branch: a flaky check in the OTA demo smoke test
Not related to the turtlebot3 profile, but it fails on this branch and on main, so it is fixed
here.
ota-demo-narrativefails on and off since 25 August, always on the same check:fault detail has >=1 environment_data snapshot. Per-topic snapshots are captured on the faultmanager capture thread pool, so they are written a bit after the fault reports CONFIRMED, and
CONFIRMED is the status the step before it waits for. The check read the fault detail once at
that moment. In the failing run on this branch it ran 124 ms before the fault manager logged
Captured 3/3 snapshotsfor the same fault code, so it read an empty list from a capture thatwas still running.
The rosbag check right below it already polls, and for the same reason. The snapshot check now
polls too, for up to 30s.
Checked against a stub API that serves the fault detail, to make sure the new form still fails
when the snapshots are genuinely missing:
The middle line is the one that matters: polling did not turn the check into one that always
passes.