Git history is getting huge due to Jupyter Notebooks

[timtreis]

Just cloned the repo and noticed that it was taking quite some time, downloading 550 MB for a fairly low-LOC repo:

I wonder if there's any merit towards switching to a fork/squash-merge approach instead of the branch based? Realistically, that history is only ever going to become larger.

Counterpoint:

• This repo is not really something many people outside devs clone so it might be overengineering.

wdyt @giovp @LucaMarconato ?

[LucaMarconato]

I wonder if what is proposed here is an acceptable solution: https://stackoverflow.com/questions/1209999/how-to-use-git-to-get-just-the-latest-revision-of-a-project.

[timtreis]

I don't think that having these extra 500 MB on disk is the issue (mildly inconvenient at best), but realistically this history will very quickly grow even larger in size if we keep working on it without squashing these Jupyter Notebook commits - same for spatialdata-notebooks.

Maybe we could implement a "squash-and-merge-only" rule to counter-act that a little bit?

[LucaMarconato]

That could be a good approach @giovp wdyt?

[giovp]

It would be interesting to decrease the size of this repo by squashing commits, not sure how to do it though. The solution you pointed to @LucaMarconato is interesting for dev perspective indeed. And also @timtreis you are right that the whole purpose of having the notebooks repo separate is that we knew this would happen in the future.

[flying-sheep]

I just went the opposite direction and moved all the tutorials to scanpy’s and anndata’s main repo because it got too annoying to deal with git submodules.

Just use blobless clones:

❯ git clone --filter=blob:none https://github.com/scverse/squidpy-tutorials.git
Cloning into 'squidpy-tutorials'...
[...]
Receiving objects: 100% (77/77), 109.72 MiB | 16.30 MiB/s, done.
Updating files: 100% (77/77), done.

[LucaMarconato]

git clone --filter=blob:none https://github.com/scverse/squidpy-tutorials.git

Good to have this in the anndata/scanpy contribution guides (if not already there).

[flying-sheep]

Done, can you take a look? scverse/scanpy#4326

(I’m going to merge it since it’s already an improvement, but if you see anything, please say so)