Firebolt: keep the three entries in the no-cold bucket - #1724
Conversation
GitHub Actions
Error
|
@alexey-milovidov, can you please take a look? |
Firebolt Core does not handle the start / stop cycle ClickHouse#1666 introduced gracefully as of Sept 2026, so those runs do not lead to meaningful measurements. Restore BENCH_RESTARTABLE=no and the no-cold tag on firebolt, firebolt-parquet and firebolt-parquet-partitioned, note at each start/stop that the cycle is off, and add a re-measured c8g.metal-48xl result taken in that configuration. The firebolt entry now pulls engine:dev_mv1, a build we verify and re-tag by hand, rather than engine:dev, which moves with every commit on our mainline. Caches are still flushed before the first run of each query, as for every other BENCH_RESTARTABLE=no system. The result files ClickHouse#1666 produced were true cold runs and keep no tag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
30d6425 to
f181af8
Compare
GitHub Actions
Waiting
How exactly? |
|
@alexey-milovidov, thanks a lot for the fast response!
Essentially, we are developing for an "always up" type of usage. E.g., certain per table metadata takes time to load -- we don't have (and do not plan on) optimizing for super-fast truly cold queries. Some of this noise creeps into the 2 warm runs as well. I think the "no-cold" bucket simply makes more sense for firebolt as of today. If this gives the default view with the combined metric an "unfair picture" of Firebolt (and other no-cold systems), you could consider this PR . I hope I'm not misunderstanding the intention of that PR, but I think this is about not counting the cold runs or the load-times for systems that don't actually measure "true cold runs" or "true load-times". Please correct me if I'm wrong. |
|
The intent was to restart every system. Even in-memory data frame systems are restarted, they just have to load the dataset from scratch for every query and count it as the cold query time. It is okay that systems have to load metadata at start. However, there is a loophole - the startup time is not counted, iirc, so lazy loading is penalized unfairly. I will give more thought to this. |
2 participants
Firebolt does not handle the start / stop cycle from #1666 gracefully as of Sept 2026, so those runs do not lead to meaningful/interesting measurements. This restores
BENCH_RESTARTABLE=noand theno-coldtag onfirebolt,firebolt-parquetandfirebolt-parquet-partitioned.Note that we continue to flush caches as all other
BENCH_RESTARTABLE=nosystems, see here.Also adds a
c8g.metal-48xlresult measured in that configuration.The result files #1666 produced were true cold runs and keep no tag; new runs will carry it again.
Additionally, switches the
fireboltentry fromengine:devtoengine:dev_mv1, a build we are verifying and re-tagging by hand, so that any mainline regressions cannot walk into the benchmark unnoticed.