Repository navigation
How should we be using dependabot in this repo? #223
Description
I noticed a few dependabot PRs in this repo, such as this one that upgrade dependencies of specific benchmarks. While in general, I think this is good practice, for a benchmark suite, I think we'd want to upgrade these dependencies as infrequently as possible to keep benchmarking results comparable with one another (and not have to always rerun baselines). Occasionally we are forced to upgrade, for example to get compatibility with a new version of CPython, but that should be deliberate.
(It's possible there is a security counterargument to be made, but I'm not a security expert and I don't know specifically whether that matters or not).
Would it make sense to update the dependabot config to only look at the top-level dependencies of pyperformance itself rather than the dependencies of specific benchmarks?
Activity
ericsnowcurrently commentedon Jul 7, 2022
+1
I don't consider the benchmarks to be a security concern, though I haven't done a thorough analysis of all the benchmark code. IIRC, none of the benchmarks access anything other than files they already own or temporary network services they start themselves.
+1
mdboom commentedon Jul 7, 2022
It looks like this may not be easily possible: dependabot/dependabot-core#4364
maurycy commentedon Sep 4, 2025
@mdboom @hugovk
I believe this can be closed now, since #407 is merged.
AA-Turner commentedon Sep 10, 2025
Seems this hasn't worked, #412 was just opened. cc @maurycy.
maurycy commentedon Sep 10, 2025
@AA-Turner
Thank you for spotting this.
I reached out to the Dependabot team directly:
since this seems to be still a very active area of development:
despite not being marked as so in the docs.
I worry that even if I clone the repository and figure out a solution, our approach won't be stable because of ongoing changes.
hugovk commentedon Sep 10, 2025
Thanks for following up with the Dependabot team. We're not in a rush to get this sorted, so can wait for them to stabilise the feature.
maurycy commentedon Sep 14, 2025
@hugovk @AA-Turner
The fix should be deployed:
Let's see how it goes before closing this issue.
🤞
maurycy commentedon Oct 26, 2025
@hugovk @AA-Turner It seems to be that we're good now?
hugovk commentedon Oct 27, 2025
I think so, we've not had any PRs for
pyperformance/data-files/since June.https://github.com/python/pyperformance/pulls?q=sort%3Aupdated-desc+is%3Apr+author%3Aapp%2Fdependabot+is%3Aclosed
Thank you!
hugovk commentedon Nov 6, 2025
We got a new one: #439
But this is out-of-schedule to fix a potential security issue, so I expect the config doesn't control this. I'll just close that one.
maurycy commentedon Jan 26, 2026
@hugovk Does #449 imply this is still a problem? 🫢
hugovk commentedon Jan 26, 2026
I think it was a security update and they're are exempt from the Dependabot config, so should be fine.