Skip to content

fix(bench): parse actual AVR package identities for Arduino Uno #1536

Description

@zackees

Context

The Arduino Uno benchmark run 36381617967 reports fbuild at 425.886 ms cold and 47.307 ms warm, faster than the Arduino CLI (824.665/462.435 ms) and PlatformIO (884.674/630.370 ms). However, it warns: fbuild build output omitted resolved package identities on Arduino Uno. The result is correctly marked unverified and its fbuild/PlatformIO ratio is excluded from publication.

The build log's AVR resolved: output identifies avr-gcc@1.70300.191015 and arduino-avr-core@5.2.0. parse_fbuild_build_packages in bench/fastled-examples/src/build_comparison.rs currently searches for toolchain-atmelavr and framework-, so it does not recognize the names fbuild actually emits and returns an empty identity map. The existing test fbuild_avr_build_log_reports_selected_package_versions uses synthetic PlatformIO-style names, so it misses the real output format.

Proposal

Teach the benchmark parser to recognize fbuild's actual AVR resolved-package output and canonicalize its AVR compiler/core package names and versions into the same comparison fields used for PlatformIO. Keep the stack-verification gate: only publish a like-for-like ratio when both tools' canonical package identities and versions match.

Acceptance criteria

  • RED → GREEN: add a focused parser test containing the exact real AVR resolved: log text from run 36381617967; demonstrate it fails with the current parser and passes after the fix, extracting avr-gcc@1.70300.191015 and arduino-avr-core@5.2.0 into canonical comparison fields.
  • A same-stack Arduino Uno fixture with matching canonical package identities and versions is marked verified and publishes its fbuild/PlatformIO ratio.
  • Missing identities or any package-version mismatch continue to mark the stack unverified and exclude the ratio.
  • The existing AVR parsing and benchmark-report tests pass.

Decisions

  • Treat this as a benchmark metadata/parser correctness bug (P2): the measured build is available, but package-identity parsing currently prevents a valid comparison from being published.
  • Normalize fbuild's names into the existing comparison schema rather than special-casing Uno in report generation, so AVR identity verification remains reusable.

Related issues

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions