URGENT: customer PII + ~40 internal files in public repo #42

Activity

1. karlwaldman

   added a commit that references this issue on Jul 3, 2026

   chore: remove internal docs and customer PII from public repo (#42) (#45

   ...

   91a8721
2. karlwaldman commented on Jul 3, 2026

   

   karlwaldman

   on Jul 3, 2026

   MemberAuthor

   More actions

   Main is clean as of PR #45 (37 files removed, all checks green). REMAINING for this issue: the history-scrub decision (git filter-repo + force-push for the two customer emails in old commits) — awaiting Karl's call, plus optionally a repo-hygiene guard note in CONTRIBUTING.md.
3. karlwaldman commented on Jul 3, 2026

   

   karlwaldman

   on Jul 3, 2026

   MemberAuthor

   More actions

   Verified STILL LIVE as of 2026-07-03 11:xx UTC. #45 removed the 37 files from HEAD, but they remain in public git history — e.g. SDK_REAL_USAGE_DATA.md (5,128 bytes) is retrievable at parent commit a730c745 via the GitHub API/git show by anyone. Repo visibility = public. So the customer-PII exposure is not yet closed; only the scrub does that.

   Ready-to-run remediation (destructive — needs your go-ahead)

   Force-push history rewrite on a public default branch is destructive (invalidates all clones/forks/open-PR diffs), so I won't run it without explicit authorization.

   # 1. Fresh mirror clone
   git clone --mirror git@github.com:OilpriceAPI/python-sdk.git && cd python-sdk.git
   # 2. Purge the 37 paths from ALL history (git-filter-repo)
   git filter-repo --invert-paths \
     --path SDK_REAL_USAGE_DATA.md --path SDK_USAGE_ANALYSIS.md \
     --path IDAN_ISSUE_GITHUB_ISSUES_CREATED.md ... (all 37 from #45)
   # 3. Force-push rewritten history
   git push --force --mirror

   Also required to fully close

   1. Rotate any credentials/tokens that appeared in the removed files (not just PII — check for API keys/secrets in the 37 files before assuming PII-only).
   2. GitHub caches commit SHAs: even after force-push, the old commits stay accessible via direct SHA URL until GitHub GC runs — open a GitHub Support ticket to purge cached views of a730c745 and the removed blobs.
   3. Assume the PII/emails were already scraped (public repo, indexed) — the affected customers may warrant notification depending on content/jurisdiction; that's a policy call.
   4. Forks: if anyone forked the repo, the PII persists in their fork independent of our scrub.

   I can execute steps 1–2 (the filter-repo + force-push) on your say-so. Want me to proceed?
4. karlwaldman

   added a commit that references this issue on Jul 3, 2026

   chore: remove internal docs and customer PII from public repo (#42) (#45

   ...

   e3b1020
5. karlwaldman commented on Jul 3, 2026

   

   karlwaldman

   on Jul 3, 2026

   MemberAuthor

   More actions

   History scrub complete (approved 2026-07-03): git-filter-repo removed the two email files, the customer-named issue doc, and test_idan_bug.py from all history. Main and 9 release tags force-pushed. Verified via GitHub commits API: zero commits now touch SDK_USER_EMAILS_SENT.md. Old clones must re-clone or reset to origin/main.
6. karlwaldman

   closed this as completedon Jul 3, 2026
7. karlwaldman commented on Jul 3, 2026

   

   karlwaldman

   on Jul 3, 2026

   MemberAuthor

   More actions

   Scrub verified ✅ — with one residual note.

   Confirmed 2026-07-03: history was rewritten (HEAD vs old a730c745 now shows status: diverged, not ancestor) and SDK_REAL_USAGE_DATA.md is gone from the current HEAD tree (404). Good — PII is out of active history.

   ⚠️ Residual (GitHub behavior, not a re-do): the orphaned commit a730c745 is still directly fetchable by SHA via the API/web (.../SDK_REAL_USAGE_DATA.md?ref=a730c745 still returns 5,128 B) until GitHub garbage-collects it. Force-push does not immediately purge dangling commits, and the SHA is publicly known (it's referenced above). To fully close: open a GitHub Support request to purge cached views of the removed blobs/commits (and confirm any credentials that were in those 37 files are rotated). Low residual risk, but worth finishing.