Repository navigation
Incremental backup skips some new events #168
Description
Activity
Dug into this one. It's not the
sinceparameter alone, it's the endpoint.--issue-eventsreads/issues/{number}/events, and cross-reference events ("mentioned in #X") never appear there, GitHub only exposes them via the timeline API. So full backups miss them too, not just incremental ones.Have confirmed with a controlled test: mentioning issue A from issue B leaves A's
updated_atuntouched, A's/eventsstays empty, and thecross-referencedevent appears only in A's/timeline;issues?since=...accordingly never lists A. Same picture on eclipse-sumo/sumo#21 from the original report:/eventsreturns 310 items all from the 2017 repo migration, while/timelinehas 2,130 items including 193 cross-references through 2024.The fix: fetch
/issues/{number}/timelineinstead of/eventsin the per-issue loop. I verified the classic event types (labeled, closed, referenced, …) come back byte-identical from both endpoints, so existingevent_dataconsumers are unaffected for those;commenteditems need filtering out (they embed full comment bodies, already covered by--issue-comments). The newly captured types (cross-referencedetc.) have their own shape — noid, asourceobject identifying the referencer — soevent_databecomes a heterogeneous list, which loose JSON consumers won't notice.One caveat that survives the fix: incremental runs select issues by
updated_at, so an issue whose only new activity is a cross-reference won't be revisited that run. It self-heals on the next touch (the per-issue fetch always rewrites the full history), so the only permanent miss is an issue that's referenced and then never touched again — worth documenting, with an occasional full run as the recommendation, rather than trying to solve.My recommendation is swap to timeline and pickup all events and live with the incremental nuance.
Happy to put together a PR along these lines with plenty of testing.
Correcting something from my earlier comment: the timeline is not a superset of the events endpoint. That held on the handful of issues I sampled, but across a full backup of this repository, 130 of 254
referencedentries appear only in/events. All of them sit on pull requests, where a commit elsewhere mentions the PR and the timeline carries the PR's owncommittedentries instead. So swapping--issue-eventsover to the timeline, which is what I suggested, would have quietly dropped half of those.#528 takes the additive route instead. A new
--issue-timelineflag stores the timeline under its owntimeline_datakey and leavesevent_dataexactly as it is, so the two are complementary rather than alternatives and both are worth running.Your diagnosis about
sincewas right too, and it needed a separate fix. A cross-reference genuinely does not change the referenced issue'supdated_at: across 178 cross-referenced issues here, not one had a matching timestamp and 45 had references dated after it. An incremental run therefore never lists the issue and never re-fetches its timeline. So on incremental runs the PR also asks GraphQL for each item's cross-reference count and newest timestamp, and re-fetches anything that disagrees with what is already stored. That costs one rate-limit point per 100 items and needs no extra state on disk.On sumo#21 specifically,
/eventsreturns 310 entries and not one is a cross-reference, while/timelinereturns 2,130 including 194 cross-references running through to this year. Those are what the backup has been missing.- added a commit that references this issue
on Jul 27, 2026
Our project https://github.com/eclipse/sumo/ has some large meta tickets like eclipse-sumo/sumo#21 where the backup does not pickup the new events anymore. When looking at the log, the issue does not even show up in the list of updated issues. This might be an upstream problem, at least I found no clear definition of what the "since" parameter in the query considers as an update and there is at least one report at https://github.community/t/github-api-how-to-get-issues-since-create-time/13680 which talks about a similar problem.