perf(gix): retain native repository handles on Repo

The native adapter previously called `gix.open_opts()` for every operation,
repeating configuration parsing and losing shared index/object-store state.
Each `Repo` now owns a native handle, associated with `Git` through a weak
reference. `close()` releases it and pickling excludes native resources.

Reuse preserves storage/environment guards. Metadata and environment changes,
and successful raw CLI launches, invalidate the configuration view and trigger
`reload()` on the retained handle. Config queries keep a separate fresh view
without synthetic safety settings. Includes reload conservatively because
GixPython does not expose their source paths. A per-repository lock prevents
concurrent refresh races; Gix handles read sharing and index/ODB refresh.

Validated Gix first: 260 affected tests and 14 subtests passed, with 3 skips;
38 native regressions passed again after the metadata guard was completed.
CLI compatibility: 222 affected tests and 14 subtests passed, with 3 skips.
Ruff lint/format, mypy and basedpyright passed.

Author: codex

doc/gix-backend.md

@@ -167,9 +167,8 @@ an import error; only an absent top-level `gix` selects CLI mode.
 operations and their complete journey on one already-open `git.Repo`, plus
 separate direct opening and discovery from `git/objects`. Fresh high-level
 wrappers preserve the cost of actual operations; imports, fixture preparation
-and parity preflight are outside timing. Native repository reopening remains
-inside operation timing, so retaining native state in the future can improve
-these measurements without changing the workload.
+and parity preflight are outside timing. Native repository refresh remains inside operation timing, so these
+measurements include the cost of keeping retained state current.
 
 At `244e418da6cc43de129cbe2908d11be4c5ad457a`, using official GixPython
 0.1.0 and the same existing CPython 3.12.14/macOS arm64 interpreter for both
@@ -208,8 +207,8 @@ ratios are observational on shared runners.
 
 ### Test-suite setup measurements
 
-Native repository reuse remains future work. The fixture optimizations below
-are implemented as separate commits, each validated with GixPython before CLI.
+The fixture optimizations below are implemented as separate commits, each
+validated with GixPython before CLI.
 The original local measurements use official GixPython 0.1.0 and existing
 CPython 3.12.14 on macOS arm64 at `89c609cf92335e75652d7725682e215a9ea080e5`.
 
@@ -219,22 +218,26 @@ expected failure. Operation reporting counted 114,800 native operations and
 39,002 CLI fallback decisions. Those decisions are not a complete subprocess
 count: raw `repo.git` calls and subprocesses started by Git are not all counted.
 
-### Retain native repository state
-
-`git._backend._repository()` calls `gix.open_opts()` for every operation;
-`git.Repo` does not retain a native repository. This deliberately avoids stale
-configuration and storage views after CLI mutations, but also loses native
-state reuse. In a 1,000-call benchmark without coverage, an object read using
-one native handle took 0.046 ms per call, compared with 0.635 ms through the
-current stream adapter. Opening and validating a native repository alone took
-0.418 ms per call. These are small repeated-read measurements, not an estimate
-of whole-suite speedup.
-
-Consider retaining native repository/object-store state per `git.Repo`, with
-explicit ownership, `close()` behavior, thread semantics, and invalidation for
-CLI/native mutations, configuration changes, and environment overrides.
-Preserve the current safety checks and diagnostics. Establish those contracts
-before replacing the current fresh-open policy.
+### Retain native repository state (implemented)
+
+Each `git.Repo` owns a native `gix.Repository`. Managed operations reuse it;
+`close()` releases it, and a later operation can reopen it. Pickling excludes
+native resources and restores the command's weak owner reference. Gix provides
+thread-safe handle access and automatic index/ODB snapshot refresh. A per-Repo
+lock serializes handle refresh, without serializing read operations.
+
+Configuration and storage metadata changes, environment changes and raw CLI
+launches cause `reload()` before reuse. Configuration queries use a separate
+fresh handle so synthetic safety settings never appear as user settings.
+Configurations with includes conservatively reload on each operation because
+the binding does not expose all included source paths. Storage overrides,
+reftable and compatibility object formats retain their existing CLI guards.
+
+An earlier 1,000-call benchmark without coverage measured an object read using
+one native handle at 0.046 ms per call, compared with 0.635 ms through the
+fresh-open stream adapter. Opening and validating a native repository alone
+took 0.418 ms per call. These small repeated-read measurements do not estimate
+whole-suite speedup.
 
 ### Reuse prepared test fixtures (implemented)
 

git/_backend.py

@@ -55,6 +55,14 @@ def _fallback(method: str, reason: str) -> Any:
 
 
 def _repository(command: Any, env: Dict[str, Any], *, query_config: bool = False) -> Any:
+    owner = command._repo() if command._repo is not None else None
+    if owner is not None:
+        with owner._gix_lock:
+            return _open_repository(command, env, query_config=query_config)
+    return _open_repository(command, env, query_config=query_config)
+
+
+def _open_repository(command: Any, env: Dict[str, Any], *, query_config: bool = False) -> Any:
     if command._git_options or command._persistent_git_options:
         raise _Unsupported("global command options")
     overrides = {**command.environment(), **env}
@@ -78,17 +86,56 @@ def _repository(command: Any, env: Dict[str, Any], *, query_config: bool = False
         options = options.config_overrides(
             ["core.fsmonitor=false", "gc.auto=0", "maintenance.auto=false", "core.hooksPath=" + os.devnull]
         )
-    repo = gix.open_opts(path, options)
+    owner = command._repo() if command._repo is not None else None
+    state = None
+    repo = None
+    if owner is not None and not query_config:
+        # Gix refreshes index/ODB snapshots itself; configuration is loaded at open.
+        paths = [
+            path,
+            os.fspath(owner.common_dir),
+            os.path.join(owner.common_dir, "config"),
+            os.path.join(path, "config.worktree"),
+        ]
+        paths += [
+            effective.get("GIT_CONFIG_SYSTEM", "/etc/gitconfig"),
+            effective.get("GIT_CONFIG_GLOBAL", os.path.expanduser("~/.gitconfig")),
+            os.path.join(effective.get("XDG_CONFIG_HOME", os.path.expanduser("~/.config")), "git", "config"),
+            os.path.join(owner.common_dir, "objects", "info", "alternates"),
+            os.path.join(path, "commondir"),
+            os.path.join(path, "gitdir"),
+        ]
+        stamps: List[Any] = []
+        for filename in paths:
+            try:
+                info = os.stat(filename)
+                stamps.append((info.st_dev, info.st_ino, info.st_size, info.st_mtime_ns, info.st_ctime_ns))
+            except FileNotFoundError:
+                stamps.append(None)
+        state = (path, tuple(sorted(effective.items())), tuple(stamps))
+        repo = owner._gix_repository
+        if repo is not None and os.path.abspath(repo.git_dir()) != os.path.abspath(path):
+            repo = None
+        if repo is not None and state != owner._gix_state:
+            repo.reload()
+    if repo is None:
+        repo = gix.open_opts(path, options)
     snapshot = repo.config_snapshot()
     if snapshot.string("extensions.refStorage") == b"reftable":
         raise _Unsupported("reftable (GIX-1)")
     if snapshot.string("extensions.compatObjectFormat") is not None:
         raise _Unsupported("compatibility object format (GIX-19)")
-    if effective.get("GIT_WORK_TREE"):
+    if effective.get("GIT_WORK_TREE") and (owner is None or query_config or state != owner._gix_state):
         workdir = effective["GIT_WORK_TREE"]
         if not os.path.isabs(workdir):
             workdir = os.path.join(command.working_dir or os.getcwd(), workdir)
         repo.set_workdir(workdir)
+    if owner is not None and not query_config:
+        owner._gix_repository = repo
+        # ponytail: includes can load arbitrary files; reopen until Gix exposes their source paths.
+        if state != owner._gix_state:
+            included = re.search(rb"\[include(?:if)?[\s\]]", repo.config_snapshot().plumbing().to_bstring(), re.I)
+            owner._gix_state = None if included else state
     return repo
 
 

git/cmd.py

@@ -638,13 +638,15 @@ class Git(metaclass=_GitMeta):
         "_git_options",
         "_persistent_git_options",
         "_environment",
+        "_repo",
     )
 
     _excluded_ = (
         "cat_file_all",
         "cat_file_header",
         "_version_info",
         "_version_info_token",
+        "_repo",
     )
 
     # Match Git's leading transport selector, including an empty helper name.
@@ -1180,6 +1182,7 @@ def __init__(self, working_dir: Union[None, PathLike] = None) -> None:
 
         # Extra environment variables to pass to git commands
         self._environment: Dict[str, Optional[str]] = {}
+        self._repo: Any = None  # Weak reference; the Repo owns native resources.
 
         # Cached version slots
         self._version_info: Union[Tuple[int, ...], None] = None
@@ -1605,6 +1608,9 @@ def execute(
             raise GitCommandNotFound(redacted_command, err) from err
         else:
             _backend.record("Git.execute", "CLI process")
+            owner = self._repo() if self._repo is not None else None
+            if owner is not None:
+                owner._gix_state = None
             # Replace with a typeguard for Popen[bytes]?
             proc.stdout = cast(BinaryIO, proc.stdout)
             proc.stderr = cast(BinaryIO, proc.stderr)

git/repo/base.py

@@ -15,7 +15,9 @@
 import shlex
 import sys
 import tempfile
+from threading import RLock
 import warnings
+import weakref
 
 import gitdb
 import gitdb.util
@@ -124,6 +126,8 @@ class Repo:
 
     # Must exist, or  __del__  will fail in case we raise on `__init__()`.
     git = cast("Git", None)
+    _gix_repository: Any = None
+    _gix_state: Any = None
 
     working_dir: PathLike
     """The working directory of the git command."""
@@ -421,7 +425,9 @@ def __init__(
                     break
 
         self.working_dir = self._working_tree_dir or self.common_dir
+        self._gix_lock = RLock()
         self.git = self.GitCommandWrapperType(self.working_dir)
+        self.git._repo = weakref.ref(self)
         self.git._environment.update(GIT_DIR=git_dir, **environment)
         if self._working_tree_dir is not None:
             self.git.update_environment(GIT_WORK_TREE=os.fspath(self._working_tree_dir))
@@ -447,6 +453,18 @@ def __init__(
                 )
             self.odb = odbt(rootpath)
 
+    def __getstate__(self) -> Dict[str, Any]:
+        return {
+            key: value
+            for key, value in self.__dict__.items()
+            if key not in ("_gix_repository", "_gix_state", "_gix_lock")
+        }
+
+    def __setstate__(self, state: Dict[str, Any]) -> None:
+        self.__dict__.update(state)
+        self._gix_lock = RLock()
+        self.git._repo = weakref.ref(self)
+
     def __enter__(self) -> "Repo":
         return self
 
@@ -460,6 +478,7 @@ def __del__(self) -> None:
             pass
 
     def close(self) -> None:
+        self._gix_repository = self._gix_state = None
         if self.git:
             self.git.clear_cache()
             # Tempfiles objects on Windows are holding references to open files until

test/test_gix_backend.py

@@ -2,6 +2,8 @@
 
 from io import BytesIO
 from pathlib import Path
+import pickle
+from concurrent.futures import ThreadPoolExecutor
 import tempfile
 from unittest.mock import patch
 
@@ -20,6 +22,49 @@ def repo(request, tmp_path):
         yield repo
 
 
+def test_native_repository_lifetime_and_refresh(repo, tmp_path):
+    handle = repo._gix_repository
+    assert isinstance(handle, gix.Repository)
+    with patch.object(gix, "open_opts", side_effect=AssertionError("unexpected reopen")):
+        assert repo.git._call_process_safe("rev_parse", "--show-object-format") == repo.object_format
+        assert repo._gix_repository is handle
+        with ThreadPoolExecutor(max_workers=2) as pool:
+            assert list(
+                pool.map(lambda _: repo.git._call_process_safe("rev_parse", "--show-object-format"), range(2))
+            ) == [
+                repo.object_format,
+                repo.object_format,
+            ]
+    Path(repo.working_dir, "file").write_text("payload")
+    repo.index.add(["file"])
+    first = repo.index.commit("native", skip_hooks=True)
+    assert repo.commit().hexsha == first.hexsha
+    repo.git.commit("--allow-empty", "-m", "CLI", "--no-verify")
+    assert repo.commit().message == "CLI\n"
+    assert repo._gix_repository is handle
+    assert set(repo.index.entries) == {("file", 0)}
+
+    with repo.config_writer() as writer:
+        writer.set_value("core", "abbrev", "9")
+    assert _backend._repository(repo.git, {}).config_snapshot().integer("core.abbrev") == 9
+    assert repo._gix_repository is handle
+    include = tmp_path / "included"
+    include.write_text("[test]\nvalue = first\n")
+    repo.git.config("include.path", str(include))
+    assert _backend._repository(repo.git, {}).config_snapshot().string("test.value") == b"first"
+    include.write_text("[test]\nvalue = second\n")
+    assert _backend._repository(repo.git, {}).config_snapshot().string("test.value") == b"second"
+
+    with pickle.loads(pickle.dumps(repo)) as restored:
+        assert restored._gix_repository is None
+        assert restored.commit().hexsha == repo.commit().hexsha
+        assert restored._gix_repository is not None
+    repo.close()
+    assert repo._gix_repository is None
+    assert repo.commit().message == "CLI\n"
+    assert repo._gix_repository is not handle
+
+
 def test_native_objects_trees_commits_and_index_reads(repo):
     actor = Actor("Example", "example@example.invalid")
     paths = ["file", "dir/with space", "dir/unicode-é", "--option"]