Skip to content

Add a Python-independent Rust API for file and notebook editing #49

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jph00 merged 1 commit into from
Sep 9, 2026
Merged

Add a Python-independent Rust API for file and notebook editing #49

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions Cargo.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,11 @@ crate-type = ["cdylib", "rlib"]

[dependencies]
crc32fast = "1"
serde = { version = "1", features = ["derive"] }
serde_json = { version = "1", features = ["arbitrary_precision"] }
tempfile = "3"
regex = "1.7.1"
pyo3 = ">=0.28"
pyo3 = { version = ">=0.28", optional = true }
tree-sitter = "0.26.12"
tree-sitter-python = "0.25.0"
tree-sitter-javascript = "0.25.0"
Expand All @@ -26,7 +29,8 @@ tree-sitter-zig = "1.1.2"
tree-sitter-swift = "0.7.3"

[features]
extension-module = ["pyo3/extension-module"]
python = ["dep:pyo3"]
extension-module = ["python", "pyo3/extension-module"]

[lints.clippy]
too_many_arguments = "allow"
Expand Down
30 changes: 23 additions & 7 deletions DEV.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,10 +14,12 @@ src/
lib.rs public API, error type, module declarations
engine.rs single- and multi-buffer edit engines producing EditResult
lnhash.rs lnhash hashing/formatting/parsing
parse.rs command parsing (script, strs, and args modes)
parse.rs compact command parsing (script and args modes)
commands.rs shared structured command fields/parser
files.rs file/cell paths, notebook JSON, views, edit/write orchestration
python.rs PyO3 bindings (incl. exhash_argv used by the CLI)
python/exhash/
__init__.py Python wrappers plus file/notebook path resolution and I/O
__init__.py Python validation, result wrappers, and display formatting
_cli.py exhash/lnhashview console-script entry points
skill.py pyskills entry point exposing exhash APIs for LLM tools
tests/
Expand All @@ -42,15 +44,25 @@ maturin develop
pytest -q
```

All tests are Python (`tests/`); there are no `cargo test` unit tests.
The existing Python API/CLI regression suite exercises the shared Rust file/cell
implementation. `cargo test` additionally tests the Rust API with no Python feature.
`cargo check --no-default-features` verifies embedding without PyO3.

## Hash verification timing

`edit_text` verifies lnhashes command-by-command immediately before each command executes. A single-line address can match the line's current hash or a recorded call-start hash from an earlier in-place edit. Records are inserted only once per line. A structural edit drops records at and below its topmost affected line; range addresses do not use the fallback.
The `$` (last line) and `%` (whole file) address forms are resolved against the current buffer and do not require hashes.
`edit_text_with_sw` exposes configurable shift width for `<` and `>`; `edit_text` defaults to `sw=4`.
In CLI and Python file-helper flows, a missing file is treated as empty input only when the parsed command set is valid against an empty buffer (for example `0|0000|a`); otherwise the original file-not-found error is preserved.
Python `file_exhash` resolves optional `path:` and notebook-cell prefixes, loads every referenced buffer, and passes the ordered buffers and commands through one `edit_buffers` binding call. Rust owns validation, buffer state, same- and cross-target transfers, printed marks, diffs, and atomic failure. Python writes successful results and preserves notebook source representation.
Rust `edit_files` resolves optional `path:` and notebook-cell prefixes, loads every
referenced buffer, and calls `edit_buffers_with_sw` once. Rust owns validation,
transfers, diffs, and file/notebook writes. Every command succeeds before writes
begin, but this is not an atomic multi-file transaction: an OS write failure can
leave earlier writes completed. Notebook source form, metadata, outputs, and
trailing newlines are preserved; each notebook is read/written once. JSON uses
arbitrary-precision numbers so unrelated notebook metadata cannot be rounded.
Python `file_exhash` and `cell_exhash` are thin adapters over this core.
File views and `file_exhash` normalize CR, CRLF, and LF line endings on read. Unicode separators remain line content, and Python result wrappers use the Rust engine's original lines so no-op detection and diffs agree. No-op file edits leave the original bytes untouched.
`lnhashview` range requests clamp `end` past EOF to the last available line, while invalid `start` values still error.

## Release
Expand Down Expand Up @@ -79,18 +91,22 @@ No local build is required for release; CI runs the release build, creates a Git

## How the CLIs work

The commands are Python console scripts declared in `[project.scripts]` (`python/exhash/_cli.py`). `exhash` and `exhash-cell` handle argument parsing, atomic file I/O, and delegate compact command parsing and editing to the extension. `lnhashview` and `lnhashview-cell` provide the corresponding address views. `exhash-open` is a fastcore `call_parse` wrapper over the document outline API.
The commands are Python console scripts declared in `[project.scripts]` (`python/exhash/_cli.py`). `exhash` and `exhash-cell` handle argument parsing and delegate compact parsing,
editing, notebook serialization, and atomic replacement to the extension. `lnhashview` and `lnhashview-cell` provide the corresponding address views. `exhash-open` is a fastcore `call_parse` wrapper over the document outline API.

## Command parsing modes

The Rust core takes commands three ways:

- Structural (PyO3 `exhash` binding): the Python wrapper validates tuple command specs and passes them through as tuples; `python.rs` builds `Command`/`Subcommand` values directly (`command_from_pyfields`), with no string round-trip. Address strings are parsed by `parse::command_from_parts`; field validation (substitute flags, transliterate counts) is shared with the compact parser via `subst_from_parts`/`translit_from_parts`. Global commands carry their subcommand as a nested tuple; text fields are verbatim, so there is no delimiter choice or escaping anywhere on this path. A trailing `.` line in an `a/i/c` payload is literal text and the binding warns about this common mistake.
- Structural (PyO3 `exhash` binding): the Python wrapper validates tuple command specs and passes them through as tuples; `commands.rs` builds `Command`/`Subcommand` values directly (`command_from_fields`), with no string round-trip. Address strings are parsed by `parse::command_from_parts`; field validation (substitute flags, transliterate counts) is shared with the compact parser via `subst_from_parts`/`translit_from_parts`. Global commands carry their subcommand as a nested tuple; text fields are verbatim, so there is no delimiter choice or escaping anywhere on this path. A trailing `.` line in an `a/i/c` payload is literal text and the binding warns about this common mistake.
- Multi-buffer structural (`edit_buffers` binding): Python passes ordered `(target, text)` buffers and target-resolved command tuples. Rust keeps one engine per target and executes the full call, including cross-target `m`/`t`, before returning per-target results.
- Compact ex-style strings, where strings are the input medium:
- `parse_commands_from_script(&str)`: for script strings; commands are separated by newlines. Single-line `a/i/c` text may be inline; if omitted, following lines up to `.` are used as the text block.
- `parse_commands_from_args(&[String], &mut BufRead)`: used by the `exhash` CLI via the `exhash_argv` binding; each arg is a command. Single-line `a/i/c` text may be inline. One command may instead read a multiline text block from stdin through EOF.

File-qualified addresses and notebook cell prefixes are resolved by the Python `file_exhash` wrapper after tuple normalization. The resulting target identifiers are opaque to Rust.
File-qualified addresses and notebook cell prefixes are resolved by `files.rs`.
The public Rust `CommandField` representation supports strings and nested arrays,
so Python and Luau use the same parser without a delimiter/string round trip.
The engine itself still treats target identifiers as opaque strings.

Commands preserve newlines in text fields. This is used by `a/i/c` payloads and by `s` pattern/replacement; replacement newlines split lines during editing. Commands without text fields do not take text. In compact strings, substitute parsing keeps Rust regex escapes intact (`\d`, `\w`, etc.) while allowing escaped command delimiters (`\/`); compact transliteration uses `y/src/dst/`. Tuple fields need no escaping at all.
186 changes: 10 additions & 176 deletions python/exhash/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
"Hash-verified line-addressed text editing. See `exhash.skill` for the workflow guide: view with `lnhashview_*` first, then edit with addresses taken from that view."

import json, re
import re
from pathlib import Path
from .exhash import line_hash as _line_hash, lnhash as _lnhash, lnhashview as _lnhashview, exhash as _exhash, edit_buffers as _edit_buffers
from .exhash import view_file as _view_file, view_cell as _view_cell, view_cells as _view_cells, edit_files as _edit_files, edit_cell as _edit_cell
from fastcore.basics import fail_clean, PrettyString

MAXLEN = 180 # Most characters shown per displayed line
Expand Down Expand Up @@ -33,7 +34,7 @@ def lnhashview(text:str, start:int=None, end:int=None) -> "LnhashView":
@fail_clean(*stdexcs)
def lnhashview_file(path:str, start:int=None, end:int=None) -> "LnhashView":
'Return lines formatted as space-padded ``lineno|hash|content`` for file at ``path`` (expands ``~``). Optional 1-based ``start``/``end`` filter the range; ``end`` past EOF is clamped.'
return LnhashView(_lnhashview(Path(path).expanduser().read_text(), start, end))
return LnhashView(_view_file(str(path), start, end))


_NOFIELD = {'d', 'p', 'j', 'sort'}
Expand Down Expand Up @@ -229,116 +230,6 @@ def _diff_out(res):



def _unescape_path(path):
out, escaped = [], False
for ch in path:
if escaped:
if ch not in ':\\': out.append('\\')
out.append(ch)
escaped = False
elif ch == '\\': escaped = True
else: out.append(ch)
if escaped: out.append('\\')
return ''.join(out)


def _split_file_prefix(s):
if _ADDR_RE.match(s): return None, s
escaped = False
for i, ch in enumerate(s):
if escaped:
escaped = False
continue
if ch == '\\':
escaped = True
continue
if ch == ':' and _ADDR_RE.match(s[i + 1:]):
path = _unescape_path(s[:i])
if not path: raise ValueError('empty filename prefix')
return _norm_path(path), s[i + 1:]
return None, s


_CELLPATH_RE = re.compile(r'(.*\.ipynb):([A-Za-z0-9_-]+)')

def _target_key(target):
path, cell = target
return path if cell is None else f'{path}:{cell}'


def _parse_fileaddr(s, default):
s = s.lstrip()
path, rest = _split_file_prefix(s)
target = (path, None) if path else default
if path and (m2 := _CELLPATH_RE.fullmatch(path)): target = (m2.group(1), m2.group(2))
m = _ADDR_RE.match(rest)
if not m: raise ValueError(f'expected exhash address near {s[:40]!r}')
return target, m.group(0), rest[m.end():]


def _parse_file_command(cmd, default):
addr, op, *fields = cmd
src, addr1, rest = _parse_fileaddr(addr, default)
has_comma, addr2 = False, None
if rest.startswith(','):
has_comma = True
src2, addr2, rest = _parse_fileaddr(rest[1:], src)
if src2 != src: raise ValueError('a range must stay within one file or cell')
if rest.strip(): raise ValueError(f'unexpected trailing characters in address: {rest!r}')
parsed = dict(src=src, addr1=addr1, addr2=addr2, has_comma=has_comma, op=op, dest=None, dest_addr=None, local=None)
local_addr = addr1 if addr2 is None else f'{addr1},{addr2}'
if op in ('m', 't'):
dest, dest_addr, tail = _parse_fileaddr(fields[0], src)
if tail.strip(): raise ValueError(f'unexpected trailing characters after destination: {tail!r}')
parsed.update(dest=dest, dest_addr=dest_addr, local=(local_addr, op, dest_addr))
else: parsed['local'] = (local_addr, op, *fields)
return parsed


_UNEXPANDED_RE = re.compile(r'\{[A-Za-z_]\w*\}|\$\{?[A-Za-z_]\w*\}?')


def _unexpanded(path):
"A note naming the IPython interpolation left literal in `path`: in a `%%exhash` line, an undefined `{name}`/`$name` is passed through as text, so the path silently becomes nonsense."
m = _UNEXPANDED_RE.search(str(path))
return f" -- note: {m.group(0)!r} looks like an unexpanded IPython variable (undefined names in a magic line are passed through literally)" if m else ""


def _load_buffer(st, target, missing_ok=False):
path, cell = target
if cell is not None:
if path not in st['nbs']:
nbp = Path(path).expanduser()
if not nbp.exists(): raise FileNotFoundError(f'notebook not found: {path}{_unexpanded(path)}')
st['nbs'][path] = json.loads(nbp.read_text())
c = _find_cell(st['nbs'][path], cell, path)
target = (path, c['id'])
if target not in st['bufs']:
text = _cell_text(c)
st['bufs'][target] = dict(target=target, path=path, cellref=c, trail_nl=text.endswith('\n'), original=text.splitlines(), lines=text.splitlines())
return st['bufs'][target]
if target in st['bufs']: return st['bufs'][target]
p = Path(path)
try: lines = p.read_text().splitlines()
except FileNotFoundError:
if not missing_ok: raise FileNotFoundError(f'file not found: {path}{_unexpanded(path)} (a new file can only be created with a 0|0000| a/i command)') from None
if not p.parent.exists(): raise FileNotFoundError(f'cannot create {path}: parent directory {p.parent} does not exist{_unexpanded(path)}') from None
lines = []
st['bufs'][target] = dict(target=target, path=path, cellref=None, original=list(lines), lines=list(lines))
return st['bufs'][target]


def _can_create_missing(parsed): return parsed['addr1'] == '0|0000|' and parsed['op'] in ('a', 'i')


def _prepare_file_command(st, parsed):
src = _load_buffer(st, parsed['src'], missing_ok=_can_create_missing(parsed) and parsed['src'][1] is None)
dest = None
if parsed['dest'] is not None:
dest = _load_buffer(st, parsed['dest'], missing_ok=parsed['dest_addr'] == '0|0000|' and parsed['dest'][1] is None)
return (_target_key(src['target']), parsed['local'], _target_key(dest['target']) if dest else None)


@fail_clean(*stdexcs)
def file_exhash(path:str, *cmds:tuple, sw:int=4, inplace:bool=True):
r'''Read files and notebook cells, apply file-aware exhash commands, and return per-target results or a combined diff.
Expand Down Expand Up @@ -371,71 +262,22 @@ def file_exhash(path:str, *cmds:tuple, sw:int=4, inplace:bool=True):
``FileSetEditResult`` is returned with ``files``, ``changed``, ``default_path``,
``res[path]`` (cell targets under ``'path:cellid'``), and ``res.format_diff(context=1)``.
'''
default, st = (_norm_path(path), None), dict(bufs={}, nbs={})
commands = [_prepare_file_command(st, _parse_file_command(cmd, default)) for cmd in _normalize_cmds(cmds)]
if not st['bufs']: _load_buffer(st, default)
by_key = {_target_key(target): buf for target, buf in st['bufs'].items()}
buffers = [(key, _text_from_lines(buf['lines'])) for key, buf in by_key.items()]
native = _edit_buffers(buffers, commands, sw=sw)
files = {key: FileEditResult(key, by_key[key]['original'], result, cell=by_key[key]['target'][1]) for key, result in native}
for key, result in files.items(): by_key[key]['lines'] = result.lines
native = _edit_files(str(path), _normalize_cmds(cmds), sw=sw, inplace=inplace)
files = {key: FileEditResult(key, result.original_lines, result, cell=cell) for key, cell, result in native}
result = FileSetEditResult(files, _norm_path(path))
if inplace:
nbs_out = {}
for t, buf in st['bufs'].items():
if buf['original'] == buf['lines']: continue
if buf['cellref'] is None: _write_lines(buf['path'], buf['lines'])
else:
new = '\n'.join(buf['lines'])
if buf['trail_nl'] and new: new += '\n'
c = buf['cellref']
c['source'] = new.splitlines(keepends=True) if isinstance(c['source'], list) else new
nbs_out[buf['path']] = st['nbs'][buf['path']]
for pth, nb in nbs_out.items(): Path(pth).expanduser().write_text(json.dumps(nb, sort_keys=True, indent=1, ensure_ascii=False) + '\n')
return result._trunc_diff()
return result




def _find_cell(nb, cell_id, path):
'The cell in `nb` whose id is ``cell_id`` (exact match or unique prefix).'
cells = [c for c in nb['cells'] if c.get('id','').startswith(cell_id)]
exact = [c for c in cells if c.get('id')==cell_id]
if exact: cells = exact
if not cells: raise KeyError(f'no cell with id {cell_id!r} in {path}')
if len(cells)>1: raise KeyError(f'cell id prefix {cell_id!r} is ambiguous in {path}')
return cells[0]


def _load_cell(path, cell_id):
'Return ``(nb, cell)`` for the cell whose id is ``cell_id`` (exact match or unique prefix).'
nbp = Path(path).expanduser()
if not nbp.exists(): raise FileNotFoundError(f'notebook not found: {path}{_unexpanded(path)}')
nb = json.loads(nbp.read_text())
return nb, _find_cell(nb, cell_id, path)


def _cell_text(cell):
src = cell['source']
return src if isinstance(src, str) else ''.join(src)
return result._trunc_diff() if inplace else result


@fail_clean(*stdexcs)
def lnhashview_cell(path:str, cell_id:str, start:int=None, end:int=None) -> "LnhashView":
'Return lines formatted as ``lineno|hash|content`` for the source of notebook cell ``cell_id`` in ipynb file at ``path`` (expands ``~``). ``cell_id`` may be an exact id or unique prefix; optional 1-based ``start``/``end`` filter the range.'
return LnhashView(_lnhashview(_cell_text(_load_cell(path, cell_id)[1]), start, end))
return LnhashView(_view_cell(str(path), cell_id, start, end))


@fail_clean(*stdexcs)
def lnhashview_cells(path:str, *cell_ids:str, start:int=None, end:int=None) -> "LnhashView":
'Return grouped lnhash views for explicit notebook cell ids in the ipynb file at ``path`` (expands ``~``). Each group starts with ``# cell <id>``; following lines keep normal ``lineno|hash|content`` format.'
out = []
for cell_id in cell_ids:
_, cell = _load_cell(path, cell_id)
out.append(f"# cell {cell.get('id', cell_id)}")
out += _lnhashview(_cell_text(cell), start, end)
return LnhashView(out)
return LnhashView(_view_cells(str(path), cell_ids, start, end))


@fail_clean(*stdexcs)
Expand All @@ -453,15 +295,7 @@ def cell_exhash(path:str, cell_id:str, *cmds:tuple, sw:int=4, inplace:bool=True)
returns the printed lines as a bare, untruncated ``lnhashview``. Pass
``inplace=False`` to preview instead: the EditResult is returned without touching the file.
"""
nb, cell = _load_cell(path, cell_id)
text = _cell_text(cell)
res = exhash(text, cmds, sw=sw)
if not inplace: return res
new = '\n'.join(res['lines'])
if text.endswith('\n') and new: new += '\n'
if new != text:
cell['source'] = new.splitlines(keepends=True) if isinstance(cell['source'], list) else new
Path(path).expanduser().write_text(json.dumps(nb, sort_keys=True, indent=1, ensure_ascii=False) + '\n')
return _diff_out(res)
res = _edit_cell(str(path), cell_id, _normalize_cmds(cmds), sw=sw, inplace=inplace)
return _diff_out(res) if inplace else res

from .outline import *
Loading