TINKERPOP-3282 TinkerStorageGraph - #3639
Draft
spmallette wants to merge 30 commits into
Draft
Conversation
TinkerStorageGraph can now durably persist committed transactions to disk through a pluggable storage engine, selected with the new gremlin.tinkergraph.storage config key with graphLocation as the storage directory. A GraphBinary write-ahead-log engine ships as the reference implementation; custom engines implement the TinkerStorage SPI. TinkerMemoryGraph is now purely in-memory: the old load-on-open / save-on-close behavior and the graphFormat key are removed. Use TinkerStorageGraph for durability or g.io() for interchange. SimpleAuthenticator now reads its credential store explicitly since the in-memory graph no longer auto-loads. Assisted-by: Claude Code:claude-opus-4-8
The pluggable storage and TinkerMemoryGraph persistence-removal entries were landing in the released beta.3 section; move them up beside the related TinkerGraph refactoring entries in the unreleased 4.0.0 section. Assisted-by: Claude Code:claude-opus-4-8
…c mode The GraphBinary storage engine's flush() only pushed bytes into the OS page cache, so an acknowledged commit could be lost on OS crash or power loss. flush() now fsyncs on commit, and compaction is made crash-safe: the snapshot is fsync'd and atomically renamed into place with directory fsyncs before the log is truncated. A new gremlin.tinkergraph.storage.sync config key selects the durability mode: 'commit' (default, fsync every commit) or 'os' (flush to the OS only; survives process crash but not power loss). Assisted-by: Claude Code:claude-opus-4-8
TinkerGraph transactions lock only their own changed elements, so commits touching disjoint elements ran their commit paths concurrently and both wrote to the storage engine's single append log at once, interleaving and corrupting its records. A fair per-graph commit-write lock now serializes the engine's persist/flush (and the compact/close paths that rewrite the same files), held only around the durable write so disjoint commits still proceed in parallel up to that point. Assisted-by: Claude Code:claude-opus-4-8
A persistent TinkerStorageGraph is a single-writer embedded store, but nothing stopped a second graph — in the same JVM or another process — from opening the same location and corrupting its log and snapshot. open() now takes an exclusive OS advisory lock on a LOCK file in the storage directory, held for the graph's lifetime and released on close(); a second open fails fast with a clear error. An OS lock is used so the kernel releases it if the JVM dies. Assisted-by: Claude Code:claude-opus-4-8
A long-running graph that is never explicitly closed grew its append log without bound, and its restart replay cost grew with it. The GraphBinary engine now tracks appended log bytes and folds the log into a snapshot on commit once it exceeds a threshold, configurable via gremlin.tinkergraph.storage.compactThreshold (default 64MB, 0 to disable). A new no-op-by-default TinkerStorage.maybeCompact SPI hook drives this, so existing engines are unaffected. Compaction runs inline under the commit lock. Assisted-by: Claude Code:claude-opus-4-8
Compaction built the entire graph snapshot as a single in-heap byte array (with a doubling buffer), so compacting a large graph needed a second full copy of it in memory and risked OOM. The snapshot is now streamed one element per framed record straight to the file, bounding peak memory to a single element. The on-disk format is unchanged: each frame is an ordinary single-entry put record that replay folds exactly as before. Write amplification (a commit rewrites each changed element in full) is documented as a known limitation rather than mitigated, since elements are small and auto-compaction bounds log growth. Assisted-by: Claude Code:claude-opus-4-8
Storage files could silently misread corrupt data: a bit-flip inside a frame went undetected, and a garbage frame length was indistinguishable from a truncated trailing append. Each frame now carries a CRC32, and every file starts with a magic + version header. On read, a complete frame with a bad CRC or a file with bad magic / an unknown version fails loudly, while a short trailing frame is still tolerated as an interrupted append. The payload allocation is bounded by the remaining file bytes, so a corrupt length can no longer trigger a huge allocation. The per-record version byte is dropped in favor of the file header. This changes the on-disk format, which is safe as the storage feature is unreleased. Assisted-by: Claude Code:claude-opus-4-8
…h storage Fills the storage test gaps left after the durability/integrity work. StorageCrashConsistencyTest reconstructs the exact on-disk states a crash leaves at each step of the write-ahead commit and compaction sequences — durable-commit-without-snapshot, snapshot-plus-log, stray temp snapshot before rename, and new-snapshot-with-log-not-yet-deleted — and asserts each reopens to the correct graph, deterministically and without killing a JVM. GraphBinaryStorageTest gains a scaled snapshot-streaming test (500 vertices + 499 edges must write exactly one frame per element) as a bounded-memory proxy, and documents why real fsync/power-loss durability is out of unit-test scope (needs OS-level fault injection). Assisted-by: Claude Code:claude-opus-4-8
The format version lived in every file header, so a store holding a snapshot and log written at different times could carry two versions that disagree. It now lives once in a store-level VERSION marker (magic + version, fsynced), the single source of truth; per-file headers keep only the magic for identity and corruption detection. Opening a store whose marker names an unsupported version fails loudly and points to g.io() export as the migration path — the engine deliberately does no in-place migration. Unknown record opcodes remain a hard error, so a genuinely incompatible change bumps the version. Assisted-by: Claude Code:claude-opus-4-8
Bring the TinkerGraph reference persistence section up to date with the storage engine's settings: the gremlin.tinkergraph.storage.sync durability modes (commit/os) and gremlin.tinkergraph.storage.compactThreshold auto-compaction knob are added to the configuration table, and the persistence prose now covers commit durability, log compaction, single-writer directory locking, and the storage-format version check with g.io() export as the migration path. Assisted-by: Claude Code:claude-opus-4-8
Move the codec-agnostic log-structured machinery — file layout, VERSION marker, CRC framing, replay fold, SyncMode durability, and crash-safe + threshold compaction — into a new abstract AbstractLogStorage. GraphBinaryStorage now supplies only the element codec through four hooks (encodeCommit, decodeFrame, writeSnapshot, beginReplay). Pure refactor: on-disk format and behavior are unchanged, so a new codec can be built by overriding the hooks alone. Assisted-by: Claude Code:claude-opus-4-8
Replace whole-DetachedVertex/DetachedEdge GraphBinary serialization with a component codec: element ids and property values go through GraphBinary's scalar serializers (a one-byte DataType tag + raw value), while labels, property keys, and meta-property keys are dictionary-encoded to small integer refs. New keys are emitted as OP_DICT_APPEND records within the same frame, before their first use. Vertex-property ids are auto-generated and no longer persisted (regenerated on load); element and edge ids are preserved. Compaction preserves dictionary numbering and writes the whole dictionary as a self-contained snapshot header, and dict-append decode is idempotent, so a log surviving the compaction crash window still resolves its refs. This removes the old whole-object codec (no dead code) and drops per-element key/label repetition and the per-property envelope, shrinking the on-disk footprint. Assisted-by: Claude Code:claude-opus-4-8
Add gremlin.tinkergraph.storage.preserveVertexPropertyIds (default false). When enabled, the GraphBinary codec persists each vertex-property id so it is stable across reopen; by default those ids are regenerated on load to keep the store smaller. A per-vertex-record flag makes each record self-describing, so a store written with the option reopens correctly even if the reader's setting differs. A new configureCodec hook lets the codec read its own config at open. Also adds codec tests: dictionary growth across log commits, dictionary rewrite on compaction, preserve-ids on/off, and a bytes/element size-regression guard (well under the ~168 bytes/element the old whole-object format cost). Assisted-by: Claude Code:claude-opus-4-8
…ption Add gremlin.tinkergraph.storage.preserveVertexPropertyIds to the TinkerGraph configuration table and note in the persistence section that element and edge ids are always preserved on reopen while auto-generated vertex-property ids are regenerated by default. Fold the compact dictionary encoding and the new option into the storage CHANGELOG entry. Assisted-by: Claude Code:claude-opus-4-8
…rips Add conformance round-trip coverage for the storage codec's breadth: a value-type matrix (int/long/float/double/boolean/byte/short/char/string/UUID/ BigInteger/BigDecimal/OffsetDateTime/Duration), collection-valued properties (List/Map/Set), null values, heterogeneous same-key types, non-Long (UUID and String) element ids, unicode keys and values, and a large-schema graph whose >127 distinct keys and >127 values under one key exercise the multi-byte varint path that small graphs never reach. Assisted-by: Claude Code:claude-opus-4-8
Add TCK cases for repeated compaction cycles, multiple open/close sessions, and concurrent commits with distinct keys. Add a crash-consistency test for the compaction crash window where a dead dictionary key forces the surviving log to diverge from the new snapshot's dictionary — guarding the decision to preserve dictionary numbering across compaction rather than renumber. Assisted-by: Claude Code:claude-opus-4-8
Surface an unknown value type code as a Corrupt storage IOException instead of an opaque NullPointerException, matching the codec's other integrity checks. Add decode-path tests for the four malformed-frame cases that clear CRC framing but are internally invalid: unknown op code, dictionary append gap, dictionary id redefinition, and unknown value type code. Assisted-by: Claude Code:claude-opus-4-8
…ty conflict Add a comment explaining why supportsConcurrentAccess() is false: a persistent TinkerStorageGraph is a single-writer store (DirectoryLock), and the feature denotes multiple connections/instances sharing the same data, not the intra-instance multi-thread transaction access the graph already provides. Re-enable the stale commented-out fail() in the concurrent meta-property test and strengthen its assertion so it proves the losing transaction rolled back, turning a decorative test into a real guard for conflict detection. Assisted-by: Claude Code:claude-opus-4-8
Rewrite the TinkerStorageGraph class Javadoc, which still described disk storage as planned future work. Add a commented, persistence-enabled server sample (tinkerstoragegraph-persistent.properties) and a Gremlin Server subsection in the persistence reference. Rename the misnamed console tinkergraph-gryo.properties to tinkergraph-storage.properties and comment the credentials sample. Assisted-by: Claude Code:claude-opus-4-8
The storage directory is now set with gremlin.tinkergraph.storage.directory, joining the gremlin.tinkergraph.storage.* settings it is only meaningful alongside. gremlin.tinkergraph.graphLocation keeps its older meaning of an interchange file and is now read only by SimpleAuthenticator, which loads the credential store from it at startup; the constant is retired from the TinkerGraph interface since no graph reads it. Also tightens the CHANGELOG entries for the storage work. Assisted-by: Claude Code:claude-opus-5
The reference framed the two implementations as a choice between memory and transactions, which left the impression that TinkerStorageGraph always writes to disk. State in the introduction and the configuration section that persistence is enabled by configuration rather than by implementation, and add a paragraph to the transactions section covering the unconfigured case, which produces no files and suits testing. Assisted-by: Claude Code:claude-opus-5
…torage A corrupt storage frame could escape the corrupt-frame contract and surface as an unchecked error instead of an IOException. readString allocated from a declared length before checking it against the record, so a small frame could demand gigabytes. readVarInt had no shift bound, so an over-long encoding wrapped and was silently accepted. Dictionary refs were dereferenced straight into the backing list, so a ref naming an undefined entry raised IndexOutOfBoundsException. All three now report corruption. Assisted-by: Claude Code:claude-opus-5
Compaction builds its snapshot by reading the graph and then discards the log, which is only sound while the graph reflects everything the log holds. That was false for as long as a changeset sat persisted but not yet applied to memory, so a compaction landing in that window snapshotted without the transaction and then deleted the record that held it, losing an acknowledged commit with no error reported. The in-memory apply now happens inside the same lock as the write to the log, and auto-compaction runs after it rather than before. Commits also become visible in the order they were recorded. Assisted-by: Claude Code:claude-opus-5
The buffer implementation backing the GraphBinary storage codec had no direct tests. Covers the primitive round-trips, the big-endian byte order the on-disk format depends on, index and capacity handling, the bulk transfer and nio views, and the bounds rejections. Assisted-by: Claude Code:claude-opus-5
An index created with createIndex() was not part of the durable state, so a graph that was reopened came back correct but with every indexed lookup silently degraded to a linear scan, with nothing to signal it. The indexed keys are now recorded beside the engine's files and the indexes are rebuilt after replay, covering data written before the restart. A dropped index stays dropped, and a record that cannot be read opens the graph with no indexes rather than failing. Definitions are kept outside the transaction log deliberately: an index only affects how fast a lookup runs and never its result, so a definition lost to a crash costs a rebuild rather than any data. The file is owned by the graph rather than the engine, so the TinkerStorage SPI is unchanged and a custom engine gets the behaviour for free. Assisted-by: Claude Code:claude-opus-5
spmallette
force-pushed
the
tinkergraph-storage
branch
from
September 10, 2026 11:40
08f6937 to
8ecdb9a
Compare
…inality TinkerGraph cardinality is graph-wide, so a persisted store holding list/set multi-properties must set gremlin.tinkergraph.defaultVertexPropertyCardinality accordingly or only the last value of each property survives a reopen. Assisted-by: Claude Code:claude-opus-4-8
…committed data Compaction iterated graph.vertices()/edges(), which expose the calling thread's uncommitted mutations, so closing a graph with an open transaction durably wrote never-committed data. writeSnapshot now reads committed container state via committedVertices()/committedEdges(), skipping uncommitted or deleted elements. Assisted-by: Claude Code:claude-opus-4-8
…growth Replay repopulated only the read-side idToKey, leaving the write-side keyToId empty, so a write session after a reopen re-appended every live key as a duplicate and the store grew on each reopen+compact cycle. decodeFrame now rebuilds keyToId alongside idToKey (first appearance wins). Assisted-by: Claude Code:claude-opus-4-8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/TINKERPOP-3282
Adds a pluggable storage option to TinkerGraph allowing it to persist data to disk as part of its transaction model. For simplicity the default storage implementation uses GraphBinary as the storage format. Review the Upgrade Documentation changes for details.