Git History Size analysis

[aditya0by0]

aditya@AuroraR10:~/python-chebai$  source /home/aditya/python-chebai-graph/.venv/bin/activate
(python-chebai-graph) aditya@AuroraR10:~/python-chebai$ ls -a
.   chebai           configs  docs  .github     LICENSE  .pre-commit-config.yaml  README.md    setup.cfg  tutorials
..  chebai.egg-info  data     .git  .gitignore  logs     pyproject.toml           .ruff_cache  tests      .vscode

(python-chebai-graph) aditya@AuroraR10:~/python-chebai$ du -sh .git
211M    .git

Yes, 211 MB for .git can be normal, but it depends heavily on the project's history.

The key point is that .git contains the entire Git history, not just the current files. If the repository has ever had large files committed—datasets, model checkpoints, generated files, logs, etc.—they can remain inside .git even after being deleted from the working tree.

(python-chebai-graph) aditya@AuroraR10:~/python-chebai$ du -h --max-depth=2 .git 2>/dev/null | sort -hr | head -30
211M    .git/objects
211M    .git
208M    .git/objects/pack
156K    .git/objects/64
72K     .git/logs
68K     .git/hooks
56K     .git/refs
56K     .git/logs/refs
40K     .git/objects/d1
36K     .git/objects/f0
36K     .git/objects/7c
32K     .git/objects/5a
32K     .git/objects/33
28K     .git/objects/c4
28K     .git/objects/9e
28K     .git/objects/94
28K     .git/objects/26
28K     .git/objects/00
24K     .git/refs/remotes
24K     .git/objects/f8
24K     .git/objects/eb
24K     .git/objects/d5
24K     .git/objects/b5
24K     .git/objects/b3
24K     .git/objects/a1
24K     .git/objects/3b
24K     .git/objects/2e
24K     .git/objects/1e
20K     .git/refs/heads
20K     .git/objects/da

(python-chebai-graph) aditya@AuroraR10:~/python-chebai$ git count-objects -vH
count: 406
size: 1.98 MiB
in-pack: 11060
packs: 1
size-pack: 207.67 MiB
prune-packable: 0
garbage: 0
size-garbage: 0 bytes

(python-chebai-graph) aditya@AuroraR10:~/python-chebai$ git rev-list --objects --all |
git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' |
awk '$1 == "blob" {print $3 "\t" $4}' |
sort -n |
tail -20
511654  tests/integration/test_data/ChEBIOver100_test/labels007.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels008.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels009.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels010.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels011.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels012.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels013.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels014.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels015.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels016.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels017.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels018.pt
511654  tests/integration/test_data/ChEBIOver100_test/labels019.pt
948368  box_results_diagrams.ipynb
1372943 outputs/zeroshot/instances_of_labels.pkl
64143950        checkpoint_flat.ckpt
65812302        checkpoint_dist.ckpt
66305742        checkpoint_ext.ckpt
73897581        outputs/zeroshot/points_with_only_extra_labels_df.pkl
81147295        outputs/zeroshot/chebi_200_transitive.pkl
(python-chebai-graph) aditya@AuroraR10:~/python-chebai$ 

Biggest historical files are:

File	Size
outputs/zeroshot/chebi_200_transitive.pkl	81 MB
outputs/zeroshot/points_with_only_extra_labels_df.pkl	74 MB
checkpoint_ext.ckpt	66 MB
checkpoint_dist.ckpt	63 MB
checkpoint_flat.ckpt	61 MB
outputs/zeroshot/instances_of_labels.pkl	1.3 MB

These files shouldn't have been commited to repository in first place. Once a commit is made for such file, git tracks the history of these file unless you reset commit. git reset --hard HEAD~1

	git revert	git reset
What it does	Creates a new commit that undoes an old commit	Moves the branch pointer backward
Existing history	Preserved	Can be rewritten
Safe on shared branches?	✅ Yes	❌ Usually no
Old commit remains in history?	✅ Yes	Depends on reset mode
Typical use	Undo a pushed commit	Undo local/unpushed commits

There what chatgpt has to say to rectify this:

Yes. You can permanently remove those files from the Git history, shrink .git, and then force-push the cleaned history to GitHub.

For your case, I recommend git-filter-repo rather than the older git filter-branch.

1. First, make a backup

Because rewriting history is destructive, make a copy of the repository:

cd ~/python-chebai

cp -a . ../python-chebai-backup

Or, even better, create a separate mirror backup:

git clone --mirror . ../python-chebai-backup.git

2. Install git-filter-repo

Check whether you already have it:

git filter-repo --help

If not:

pip install git-filter-repo

3. Add these files to .gitignore

If these are generated checkpoints/results that shouldn't be committed again:

cat >> .gitignore <<'EOF'

# Generated model checkpoints
*.ckpt

# Generated outputs
outputs/
EOF

Then commit that change:

git add .gitignore
git commit -m "Ignore generated checkpoints and outputs"

4. Permanently remove them from ALL history

From the repository root:

git filter-repo --force \
  --path checkpoint_flat.ckpt \
  --path checkpoint_dist.ckpt \
  --path checkpoint_ext.ckpt \
  --path outputs/zeroshot/chebi_200_transitive.pkl \
  --path outputs/zeroshot/points_with_only_extra_labels_df.pkl \
  --path outputs/zeroshot/instances_of_labels.pkl \
  --invert-paths

This is the important step.

It doesn't merely delete the files from your current checkout. It removes those paths from every commit in the rewritten history.

5. Check the result

Run:

git count-objects -vH

and:

du -sh .git

You should see a substantial reduction from your current:

211M    .git
207.67 MiB size-pack

You can also verify that the files are gone from history:

git rev-list --objects --all | grep -E 'checkpoint_(flat|dist|ext)\.ckpt|outputs/zeroshot/(chebi_200_transitive|points_with_only_extra_labels_df|instances_of_labels)\.pkl'

Ideally, that produces no output.

6. Clean up the old Git objects

After rewriting history:

git reflog expire --expire=now --all
git gc --prune=now --aggressive

Then:

du -sh .git

This is when the old objects should actually disappear from your local .git.

7. Push the rewritten history to GitHub

⚠️ This is the dangerous part.

Because you've rewritten commit history, a normal push will be rejected.

First check your remote:

git remote -v

Then push:

git push --force-with-lease --all
git push --force-with-lease --tags

I prefer --force-with-lease over --force because it provides some protection against accidentally overwriting someone else's newer work.

---

One important warning

If this is a repository that other people are actively working on, don't do this without coordinating with them.

After the force push, their existing clones will contain the old history. They'll generally need to re-clone or carefully reset/rebase their local branches.

If you're the only person working on this repository, this is much simpler and is generally safe after making the backup.

Also, GitHub's copy of the old objects may not disappear immediately just because you force-push. The repository's visible history will be rewritten, but GitHub may retain unreachable objects temporarily before garbage collection.

Given your output, I would absolutely clean these files out of Git history if they're generated checkpoints/results. The three .ckpt files alone are ~190 MB, and the two large .pkl files add another ~155 MB.