Skip to content

Commit 6881712

Browse files
codexByron
authored andcommitted
refactor!: delegate index state and metadata decoding to Git and Gix
Python previously retained mutable index dictionaries and reconstructed index files from those entries. That duplicated Git semantics and allowed validation and publication to diverge. Keep deferred and virtual indexes as opaque bytes; materialize a private file before Git edits it, verify requested entries survived, and publish through the existing atomic lock. Replace `IndexFile.entries` with `iter_entries()` and `entry()` and migrate submodule, tutorial and test callers. Preserve deferred add/remove and conflict resolution, alternate publication, index versions and flags, split snapshots, checkout/diff visibility and hook edits. Failed ordinary edits retain pending and published state. Reject NULs before encoding stdin records. Keep source containment checks because Python reads those paths before Git validates the recorded destination. Use GixPython's structured commit/tag decoders, including signature headers and encoding, instead of parsing native object bytes. Retain raw CLI commit/tag metadata decoding where formatted Git output cannot preserve those fields. Delegate textual date syntax and trailer discovery to Git. Record Gix's unsafe index insertion, missing date parser and missing strict tag validation as `GIX-26` through `GIX-28`; private index edits and tag storage use CLI fallbacks. A native byte round trip alone does not prove Git-compatible validation. Remove the deprecated `GitDB` selection, API aliases, global shell switch, automatic environment expansion and Windows error-hiding switches. Keep explicit shell selection and tilde expansion. Document replacements and changed timezone and trailer behavior in `doc/source/changes.rst`. Delete obsolete parser/deprecation tests, repeated rejection matrices and historical timing-only tests. Retain happy paths, Python glue/security failures, atomicity and demonstrated compatibility regressions. The maintained benchmark continues to check parity and CLI process budgets. Validation on macOS with Git 2.54.0 and GixPython 0.1.0: - Final retained Gix suite: 1,767 passed, 51 skipped, one expected failure, and 44 subtests passed. CLI suite: 1,672 passed, 52 skipped, one expected failure, and 44 subtests passed. The new tag rejection test passed separately on CLI after that suite collected; final framing/fixture checks passed on both backends (17 tests each). - All five cached downstream profiles passed on both backends: Bandit, DataHub, LangChain, MLflow and SWE-bench. - All 14 benchmark result digests matched and existing Gix CLI ceilings passed. Fast timings ran alongside validation and do not establish a speed change. - `pre-commit`, `ruff`, `mypy`, `basedpyright`, the warnings-as-errors Sphinx build, and `git diff --check` passed. Windows/Cygwin and the minimum Git 2.52 were not exercised for this migration. Removed APIs and corrected date/trailer behavior require caller migration. Cygwin CI's former `perf` matrix entry collected no tests and exited 5 after the historical timing tests were removed; the remaining comparator module skips without the optional `pyperf` dependency. Collapse the obsolete split into one full retained Cygwin suite. The dedicated backend benchmark workflow continues to install benchmark dependencies and enforce parity/CLI budgets. The applicable workflow pre-commit checks and `git diff --check` passed; Cygwin execution is verified by hosted CI. The Windows Gix job exposed a platform assumption in the new literal-path regression: `os.path.expanduser` uses `USERPROFILE` on Windows rather than `HOME`. Set both fixture variables to the temporary home, preserving the actual tilde-expansion and literal environment-syntax checks. Both corrected fixture tests passed with the CLI and Gix installations and an absolute Git executable. An `ntpath.expanduser` probe verified the Windows home variable; relevant pre-commit hooks and `basedpyright --warnings` also passed.
1 parent 46ba8ff commit 6881712

57 files changed

Lines changed: 879 additions & 3128 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.basedpyright/baseline.json‎

Lines changed: 0 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -228,24 +228,6 @@
228228
}
229229
}
230230
],
231-
"./git/remote.py": [
232-
{
233-
"code": "reportAttributeAccessIssue",
234-
"range": {
235-
"startColumn": 26,
236-
"endColumn": 38,
237-
"lineCount": 1
238-
}
239-
},
240-
{
241-
"code": "reportAttributeAccessIssue",
242-
"range": {
243-
"startColumn": 26,
244-
"endColumn": 38,
245-
"lineCount": 1
246-
}
247-
}
248-
],
249231
"./git/repo/base.py": [
250232
{
251233
"code": "reportTypedDictNotRequiredAccess",
@@ -311,16 +293,6 @@
311293
"lineCount": 1
312294
}
313295
}
314-
],
315-
"./test/deprecation/test_basic.py": [
316-
{
317-
"code": "reportUnusedExpression",
318-
"range": {
319-
"startColumn": 12,
320-
"endColumn": 62,
321-
"lineCount": 1
322-
}
323-
}
324296
]
325297
}
326298
}

‎.github/workflows/cygwin-test.yml‎

Lines changed: 2 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -11,17 +11,6 @@ permissions:
1111

1212
jobs:
1313
test:
14-
strategy:
15-
matrix:
16-
selection: [fast, perf]
17-
include:
18-
- selection: fast
19-
additional-pytest-args: --ignore=test/performance
20-
- selection: perf
21-
additional-pytest-args: test/performance
22-
23-
fail-fast: false
24-
2514
runs-on: windows-latest
2615

2716
env:
@@ -149,6 +138,6 @@ jobs:
149138
python --version
150139
python -c 'import os, sys; print(f"{sys.platform=}, {os.name=}")'
151140
152-
- name: Test with pytest (${{ matrix.additional-pytest-args }})
141+
- name: Test with pytest
153142
run: |
154-
pytest --color=yes -p no:sugar --instafail -vv ${{ matrix.additional-pytest-args }}
143+
pytest --color=yes -p no:sugar --instafail -vv

‎doc/gix-backend.md‎

Lines changed: 36 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,36 @@ provide general Git-path resolution. A `native` counter must represent a
4242
Gix-backed operation, and backend timings include its Python glue and any CLI
4343
fallbacks.
4444

45+
## Index and object metadata lifecycle
46+
47+
`IndexFile` queries immutable entry records; it does not retain a parsed index.
48+
Deferred/virtual state consists of opaque file bytes, materialized before Git
49+
operates and read back only after success. Private edits use Git because native
50+
bindings cannot load an arbitrary existing index (GIX-3) or validate a checked
51+
batch of paths (GIX-26). Split snapshots become standalone through Git and regain
52+
split publication through Git. Native default-index queries, fresh v2 indexes
53+
and verified tree construction remain supported. Backend coverage must count
54+
these editing fallbacks.
55+
56+
Commit metadata uses `Repository.find_commit().decode()` plus structured
57+
`Commit.author()`/`committer()` signatures. Tag metadata uses
58+
`Repository.find_tag().decode()` and `Tag.tagger()`. Decoded extra headers retain
59+
`gpgsig`. Raw Gix object bytes are used only for object streams. If repository
60+
capabilities require CLI fallback, metadata uses `git cat-file`: general formatted
61+
Git queries cannot faithfully expose all signature headers or arbitrary tags.
62+
The bounded public identity helpers remain Python glue. Human date parsing uses
63+
Git until GIX-27 is exposed; trailer extraction already uses Git.
64+
65+
Benchmark remeasurement on 2026-10-09 used the same fixed SHA-1/files fixture
66+
`6ba2c0a2f9ee7feffd7e079621c4845820180c9a`. All 14 result digests match between
67+
CLI and Gix. Gix still uses 1 CLI process for the complete read journey, 0 for
68+
index reads and commit history, 1 for patch diffs/opening, and 3 for discovery.
69+
The CLI journey now uses 73 launches, including 26 for the 25-commit history;
70+
raw metadata queries preserve exact bytes. Existing Gix budgets remain valid.
71+
The `--fast` timing samples ran alongside validation and have substantial jitter;
72+
use their digests/counts, not their wall times, as evidence for this migration.
73+
These read-only budgets make no claim that index editing is native.
74+
4575
## Install and test without package indexes
4676

4777
The prepared environments in this checkout are `.venv` (CLI) and `.tox/gix`
@@ -186,8 +216,8 @@ repository-format validation gap.
186216
| `Commit.iter_items`, `Repo.iter_commits` | First-parent walks and a single tip | General history ordering; GIX-8 |
187217
| Index entry reads, `ls_files` | Stage/mode/OID/path and exposed index flags | Custom and sparse indexes; GIX-3/4 |
188218
| `IndexFile.version`, `update_index` query | Native index version | Actual `update-index` mutations |
189-
| `IndexFile.write` and index persistence | Edit a private native index, publish through the existing lock | Custom, sparse, split, non-v2, unmerged, overlapping, or null-ID entries; GIX-3/4/13 |
190-
| ODB `store`, managed `hash_object` | Native object hashing/writing with byte-fidelity preflight | Large/nonseekable input and changed serialization; GIX-2/5 |
219+
| Index editing and `IndexFile.write` | Opaque byte snapshots; atomic locked publication | Private-file edits always use Git: arbitrary native index loading and strict pathname validation are missing; GIX-3/4/13/26 |
220+
| ODB `store`, managed `hash_object` | Native object hashing/writing with byte-fidelity preflight | Tags, large/nonseekable input and changed serialization; GIX-2/5/28 |
191221
| `IndexFile.write_tree` | Native tree editor with child-kind and null-ID validation | Missing non-gitlink children, invalid entries; GIX-7/24 |
192222
| Tree serialization, `mktree` | Native tree editor with child-kind and null-ID validation | Missing children, unsupported or invalid entries; GIX-7/24 |
193223
| Fresh index preparation, `read_tree` | Empty index or index from one tree | Existing index, merges, non-v2 selection; GIX-13 |
@@ -738,6 +768,10 @@ identify the historical fixture where needed.
738768
| GIX-23 | Missing API | No hook lookup or execution API is bound in GixPython 0.1.0. The removed absence fast path read config through Gix but used Python `os.stat()` to declare success. | Bind native hook lookup with configured/default path, linked-worktree, missing/nonexecutable-hook and diagnostic semantics, plus execution where needed. Until then all managed hook calls use Git, including `--ignore-missing` no-ops. A Python filesystem check is not a Gix implementation. |
739769
| GIX-24 | Bug | In both object formats, `new_commit_as()` accepts a blob as the tree or a parent where `git commit-tree` rejects it. Tree-editor `upsert()`/`write()` accepts object IDs whose actual kinds disagree with blob/tree/gitlink modes; `git mktree --missing` rejects all three tested mismatches. `edit_references_as()` accepts a blob target under `refs/heads/`, rejected by `git update-ref`. | Provide checked commit/tree construction and reference edits, at least in Git-strict mode. Match Git's kind and direct-branch target validation before writing and under the required ref locks, including its distinct handling of symbolic aliases. Existing `_commit_tree`, `_write_tree` and `_update_ref` guards query Gix headers and select CLI on invalid inputs; they do not replace Gix writes with Python. |
740770
| GIX-25 | Bug; missing API | After `git symbolic-ref refs/heads/dangling refs/heads/missing`, `Repository.references().all()` enumerates the dangling name, while `git for-each-ref --format=%(refname)` omits it. Valid symbolic aliases must remain present. Both SHA-1 and SHA-256 probes reproduce this difference. | Expose Git-compatible enumeration with the same dangling-reference and diagnostic behavior, retaining the raw iterator for callers that need it. `_for_each_ref` forces Gix to resolve symbolic targets and falls back to Git on failure; keep that fallback until a compatible Gix API/mode is verified. |
771+
| GIX-26 | Bug; missing API | `dangerously_push_entry(IndexStat(), oid, 0, 0o120000, b".gitmodules")`, followed by `sort_entries()`, `verify_entries()` and `write()`, succeeds and retains the unsafe symlink. Protected Git `update-index -z --index-info` ignores it with exit 0 and `Ignoring path .gitmodules`; protected `read-tree` rejects it. The native fresh `index_from_tree` also rejects it. | A Git-strict checked index batch editor that validates names/modes, preserves metadata and reports rejected records. `verify_entries()` verifies ordering, not pathname safety. All private-index edits use Git with `core.protectHFS=true` and `core.protectNTFS=true`, followed by requested-entry readback; no Python dictionary reconstruction or native dangerous insertion remains. |
772+
| GIX-27 | Missing API | `gix.Time` exposes seconds/offset but no human date parser in GixPython 0.1.0. | Bind Git-compatible textual date parsing with timezone semantics. `parse_date()` uses `git var GIT_AUTHOR_IDENT` with fixed identity and per-command date, then adapts the normalized timestamp/offset. |
773+
| GIX-28 | Bug; missing API | GixPython 0.1.0 on Gitoxide `f819565c2c4c56619c4888acef6cf3b8144cbccb` stores `object <blob>\ntype blob\ntag invalid\n\nmessage\n` through `write_object("tag", data)`. Git 2.54.0 rejects the same payload through `hash-object -t tag -w --stdin` with `missingTaggerEntry`. Byte-preserving native readback does not establish Git's validation. | Expose a Git-strict tag writer or validation API. Managed tag hashing/storage uses Git before any native write; `test_tag_storage_rejects_missing_tagger` verifies failure and absence from storage. Keep the bug open until upstream validation is verified. |
774+
741775

742776
Windows reproduction for GIX-14 uses CPython 3.12.13, Git
743777
2.55.0.windows.3 and the released GixPython 0.1.0 source distribution with

‎doc/source/changes.rst‎

Lines changed: 50 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,10 @@ API changes
5151
* ``GitCmdObjectDB`` no longer inherits ``LooseObjectDB``. Its object reads,
5252
writes, existence checks, and enumeration use Git, including packed objects.
5353
Precompressed input streams and custom object-output writers are unsupported.
54-
The deprecated ``GitDB`` remains explicitly selectable with its existing
55-
warning and limitations; ``gitdb`` remains a dependency for shared types.
54+
``GitDB`` exports and backend selection are removed. ``odbt`` accepts only
55+
``GitCmdObjectDB`` and its subclasses, rejecting unsupported types before
56+
initialization or cloning. Remove ``odbt=GitDB`` or select ``GitCmdObjectDB``.
57+
``gitdb`` remains a dependency for shared types and utilities.
5658
* ``Repo.object_format`` and ``Repo.ref_format`` report Git's storage formats.
5759
``Repo.alternates`` is read-only and reports effective absolute alternate
5860
directories, including environment and transitive alternates. Direct editing
@@ -99,6 +101,52 @@ API changes
99101
Fetch/pull results are derived from command output rather than ``FETCH_HEAD``
100102
file parsing. Revision strings follow Git's native revision grammar.
101103

104+
* ``IndexFile.entries`` is removed. Use ``iter_entries()`` for immutable query
105+
records, ``entry(path, stage=0)`` for exact lookup (missing entries raise
106+
``KeyError``), ``add()``, and ``remove()`` for edits. Python retains only opaque
107+
index-file bytes for deferred/virtual indexes. Git edits private files and
108+
validates readback before atomic publication. ``add(write=False)`` and
109+
``resolve_blobs()`` retain deferred edits; ``remove(..., write=False)`` now does
110+
too, unless ``working_tree=True``. ``write(file_path)`` publishes to an alternate
111+
path. Checkout and diff use pending bytes. Git owns versions, flags, sparse and
112+
split indexes; deferred split snapshots are made standalone through Git.
113+
Ordinary failed index edits preserve pending and published state. Worktree,
114+
object storage and hook side effects are not rolled back. Commit hooks see the
115+
materialized index; their index edits survive failures without advancing HEAD.
116+
Binary-parser tests and duplicate filename-rejection matrices are replaced by
117+
happy paths, Python glue failures and representative compatibility regressions.
118+
Historical timing-only tests are removed; the maintained backend benchmark
119+
checks result parity and CLI budgets.
120+
* Commit/tag metadata uses structured Gix decoders when the Gix backend supports
121+
the repository. The CLI backend retains raw decoding because Git has no faithful
122+
formatted query for all commit headers or arbitrary tag objects. ``gpgsig``
123+
remains readable. The small ``Actor.from_string()`` and
124+
``parse_actor_and_date()`` identity helpers remain supported.
125+
* ``parse_date()`` delegates text syntax and timezone interpretation to Git;
126+
aware datetimes retain direct conversion and invalid inputs raise ``ValueError``.
127+
ISO/RFC dates now apply their timezone to the UTC timestamp, correcting the
128+
previous behavior. Dates without a zone use Git's local timezone, and accepted
129+
date syntax follows the installed Git version. Use an explicit timezone for
130+
reproducible results. ``co_authors`` now uses Git's final trailer block rather
131+
than scanning arbitrary message lines; use a valid trailer block after a blank
132+
line.
133+
* Removed deprecated APIs: ``Git.USE_SHELL`` (explicit ``Git.execute(shell=...)``
134+
remains), ``Diff.renamed`` (use ``renamed_file``), ``Commit.trailers`` (use
135+
``trailers_list`` or ``trailers_dict``), ``Actor.name_email_regex`` (use
136+
``Actor.from_string``), ``git.util.Iterable`` (use ``IterableObj``),
137+
``git.compat.is_win/is_posix/is_darwin`` (use ``os.name``/``sys.platform``), and
138+
``git.types.Lit_commit_ish`` (use ``Literal["commit", "tag"]`` or
139+
``GitObjectTypeString``). Top-level typing exports and private module aliases
140+
are removed; import from ``typing`` or the owning module. ``git.util`` now
141+
exposes the actual utility module. Abstract ``Traversable.traverse`` and
142+
``list_traverse`` raise ``NotImplementedError``; use concrete implementations.
143+
* Paths and URLs no longer expand ``$VAR``/``%VAR%`` automatically. Removed
144+
``expand_vars`` switches from ``Repo``, ``Repo.init``, ``expand_path``,
145+
``cygpath`` and ``Git.polish_url``. Expand explicitly with
146+
``os.path.expandvars`` if wanted; initial ``~`` expansion remains.
147+
``HIDE_WINDOWS_KNOWN_ERRORS`` and ``HIDE_WINDOWS_FREEZE_ERRORS`` are removed.
148+
``rmtree`` propagates filesystem errors rather than raising ``SkipTest``.
149+
102150
3.2.1
103151
=====
104152

‎doc/source/intro.rst‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ Overview / Install
66

77
GitPython is a python library used to interact with git repositories, high-level like git-porcelain, or low-level like git-plumbing.
88

9-
It provides Python objects for repository data and delegates Git operations to the Git executable. The default backend supports both SHA-1 and SHA-256 object IDs and both files and reftable reference storage. The legacy pure-Python GitDB backend remains available but is deprecated.
9+
It provides Python objects for repository data and delegates Git operations to the Git executable. The default backend supports both SHA-1 and SHA-256 object IDs and both files and reftable reference storage. The legacy pure-Python GitDB backend is removed.
1010

1111
The object database implementation is optimized for handling large quantities of objects and large datasets, which is achieved by using low-level structures and data streaming.
1212

@@ -15,7 +15,7 @@ Requirements
1515

1616
* `Python`_ >= 3.8
1717
* `Git`_ 2.52 or newer
18-
* `GitDB`_ - shared data types and the deprecated legacy object database
18+
* `GitDB`_ - shared data types and utilities
1919
* `typing_extensions`_ >= 3.7.3.4 (if python < 3.10)
2020

2121
.. _Python: https://www.python.org

‎doc/source/tutorial.rst‎

Lines changed: 4 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -341,7 +341,7 @@ As trees allow direct access to their intermediate child entries only, use the t
341341

342342
The Index Object
343343
****************
344-
The git index is the stage containing changes to be written with the next commit or where merges finally have to take place. You may access and manipulate semantic entries (mode, object ID, path, and stage) using the :class:`IndexFile <git.index.base.IndexFile>` object. Git reads and writes the underlying index; raw stat fields and binary index extensions are not exposed.
344+
The git index is the stage containing changes to be written with the next commit or where merges finally have to take place. Use :meth:`IndexFile.iter_entries <git.index.base.IndexFile.iter_entries>` and :meth:`IndexFile.entry <git.index.base.IndexFile.entry>` to query immutable entry records (mode, object ID, path, and stage); use ``add`` and ``remove`` to edit them. Deferred edits retain opaque index-file bytes. Git reads and writes the underlying index; raw stat fields and binary index extensions are not exposed.
345345
Modify the index with ease
346346

347347
.. literalinclude:: ../../test/test_docs.py
@@ -522,20 +522,9 @@ resolves abbreviated object IDs through persistent ``git cat-file`` processes::
522522
# Equivalent explicit selection:
523523
repo = Repo("path/to/repo", odbt=GitCmdObjectDB)
524524

525-
GitDB
526-
=====
527-
.. warning::
528-
The pure-Python ``GitDB`` backend is deprecated due to security and performance
529-
issues. Its object parsers can exhaust resources or return incorrect object
530-
data when processing untrusted repositories. Do not use it for untrusted data.
531-
532-
Selecting ``odbt=GitDB`` (including a subclass) emits a ``DeprecationWarning``.
533-
To migrate, remove ``odbt=GitDB`` or replace it with ``odbt=GitCmdObjectDB`` when
534-
opening, initializing, or cloning a repository. The deprecated backend remains
535-
available for compatibility; deprecation does not fix its parsing issues.
536-
537-
The ``gitdb`` package remains a dependency because GitPython still uses its shared
538-
types and utilities.
525+
Only ``GitCmdObjectDB`` and subclasses can be selected through ``odbt``. The
526+
legacy ``GitDB`` backend is removed. The ``gitdb`` package remains a dependency
527+
for shared types and utilities.
539528

540529
Git Command Debugging and Customization
541530
***************************************

0 commit comments

Comments
 (0)