Skip to content

feat(rows): add persistent UUIDv7 identifiers and UUID-based API access - #3229

Open
foolfitz wants to merge 5 commits into
nextcloud:mainfrom
foolfitz:feat/3093-row-uuid-squashed
Open

foolfitz wants to merge 5 commits into
nextcloud:mainfrom
foolfitz:feat/3093-row-uuid-squashed

Conversation

@foolfitz

@foolfitz foolfitz commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Related issue: #3093

Add a persistent UUIDv7 to each row so that it can still be identified after export and import through user migration, even if its integer ID changes. The existing integer primary keys and cell associations remain unchanged.

Main changes

  • Store UUIDv7 values in tables_row_sleeves and add a unique index covering all rows across the instance. Once assigned, a UUID cannot be replaced or cleared.
  • Assign UUIDs to new rows when they are inserted. Backfill existing rows through a resumable background job to avoid processing all historical rows synchronously during the schema migration. During backfill, UUIDs may be null, and rows remain accessible by numeric ID.
  • Preserve row UUIDs in user migration archives. On import, retain a UUID if it is unused on the destination. If a collision occurs, generate a new UUID for the imported copy without overwriting the existing row. Generate new UUIDs for older archives that lack them, and reject imports containing invalid UUIDs.
  • Add a single-row GET endpoint at /api/2/{tables|views}/{nodeId}/rows/{rowId} in API v2, and allow GET, PUT, and DELETE to accept either an integer ID or a UUID. UUID lookups retain the existing permissions, view filters, and column restrictions. Malformed identifiers return 400; rows not found within the authorized scope return 404. Add uuid to row responses and update the OpenAPI and TypeScript definitions together.

The UUID API provided by this PR can read, update, or delete rows on the instance receiving the request. For example, after importing data from Site A into Site B, UUID requests sent to Site B operate only on Site B's data and are not forwarded to Site A.

This defines the scope of this PR. Work left for later includes federated UUID forwarding, UUID addressing on public share token routes, cross-site synchronization, and full NOT NULL enforcement. UUID preservation applies to user migration archives; CSV import does not yet support migrating rows while retaining their source UUIDs.

Regarding the database migration: before merging, confirm with the maintainers which Tables release will include this change and align the migration's version identifier accordingly. This ensures that upgrading an existing installation automatically adds the UUID column and index and schedules background backfill for existing rows.

Testing

Manual testing: preserving UUIDs after archive migration

Using version 0e70ed8c, data with two text columns, Item and Status, was migrated through a User migration archive between two independent Nextcloud 33.0.8 / SQLite environments, with Site A as the source and Site B as the destination.

1. Create data on Site A and record the original ID and UUID

A row containing Transfer notebook / keep was created, and its identifiers were queried. The request returned HTTP 200, integer ID 4, and UUID 01a1ab40-6328-75f1-8976-3cd3e2a98dd8.

M2-source-id-uuid

2. Export a user migration archive from Site A

The user:export --types=tables command exported the Tables data, and the terminal reported a successful export. A separate inspection of the archive confirmed that it preserved the row's UUID and original content.

M3-export-completed

3. Import the same archive into Site B

The user:import command imported the archive into a dedicated empty account on Site B. The terminal reported that the Tables import completed, with the data written to Site B's own database.

M4-import-completed

4. Query the source row on Site A using the original UUID

A GET request using the original UUID was sent from Site A's browser session. It returned HTTP 200, with integer ID 4 and the same UUID as before export.

M5-source-live-get

5. Query the imported row on Site B using the same UUID

A GET request using the same UUID was sent from a separate browser session on Site B. It returned HTTP 200. The integer ID had been reassigned to 108, while the UUID still matched Site A's and the content remained Transfer notebook / keep. This confirmed that the original UUID could still locate the data after migration.

M5-destination-live-get

6. Update the content on Site B using the UUID

A PUT request using the UUID path changed Item to Notebook after transfer. It returned HTTP 200, with integer ID 108, an unchanged UUID, and Status still set to keep. The table in the screenshot had not yet been reloaded; the PUT response confirms the update.

M6-put-by-uuid

7. Query Site B again to confirm that the update was saved

A separate GET request using the UUID returned HTTP 200, with Item set to Notebook after transfer and both the integer ID and UUID unchanged. The reloaded table also displayed the new value.

M6-update-after-transfer

8. Query Site A again to confirm that the source data was unaffected

A final GET request to Site A using the original UUID returned HTTP 200, with integer ID 4, the same UUID, and the original content, Transfer notebook / keep. This confirmed that Site B's update had not synchronized back to Site A.

M6-source-unchanged

This manual test covers archive migration and local UUID GET/PUT operations. It does not cover DELETE, UUID collisions, historical backfill, view-sharing permissions, or federation operations.

Automated verification and upgrade measurements

Local verification records show that the full PHPUnit suite, MySQL/MariaDB/PostgreSQL database tests, Psalm, coding standards, PHP syntax checks, and TypeScript checks all passed. Regenerating the OpenAPI specification and types produced no differences.

An earlier SQLite benchmark with 1,000,014 rows recorded approximately 1.44 seconds for the schema/index migration and 41.48 seconds of total background worker time. All 165 sampled HTTP requests during backfill returned 200. This measurement predates the final cursor-write adjustment and has not been repeated; total worker time does not represent elapsed completion time under the normal cron schedule.

🏁 Checklist

  • ⛑️ Tests (unit and/or integration) are included or not needed
  • 🔙 Backport requests are created or not needed: /backport to stableX.X
  • 📅 Milestone is set
  • 🌸 PR title is meaningful (if it should be in the changelog: is it meaningful to users?)

🤖 AI (if applicable)

  • The content of this PR was partly or fully generated using AI

Codex assisted with developing the code and tests, and Claude Code assisted with their review. I performed the manual migration test and captured the screenshots; Codex assisted with drafting the PR description.

@foolfitz
foolfitz marked this pull request as ready for review November 6, 2026 07:02
@foolfitz foolfitz changed the title feat(rows): add persistent UUIDv7 identifiers and UUID-based API access #3093 feat(rows): add persistent UUIDv7 identifiers and UUID-based API access Nov 6, 2026
@enjeck enjeck mentioned this pull request Nov 18, 2026
4 of 5 tasks

@blizzz blizzz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart of the comments, we get some more overhead on PUT and DELETE requests:

(RowOCSController.php:190,241). It checks the read permission, loads the view and runs a filtered scope query. Then updateRowData/delete do all of that again: getRowById, the permission checks, and isRowInViewPresent (RowService.php:707,831).

(by code, without benchmarking)

I would prefer keeping the UUID out of the core logic as much as possible. Handle them on the Controller parts, where necessary, and continue working with the ID. The main way to interact with Tables should still be the id.

All in all there are too many other things creeping into this PR, and the changes are more complicated than necessary.

use OCP\Migration\IOutput;
use OCP\Migration\SimpleMigrationStep;

class Version2500Date20261103000000 extends SimpleMigrationStep {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When adding a new migration, also bump the version in https://github.com/nextcloud/tables/blob/main/appinfo/info.xml#L29, now to 2.4.1-dev.0. Otherwise this will not be picked up on anything that uses existing setups.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, with one adjustment: main moved to 2.5.0-dev.0 when #3131 was merged today, so the PR now bumps to 2.5.0-dev.1 (in package.json too), and the migration is named Version2500Date20261120000000 to sort after main's newest one.

Comment thread lib/Controller/RowOCSController.php Outdated
#[NoAdminRequired]
#[RequirePermission(permission: Application::PERMISSION_READ, typeParam: 'nodeCollection')]
#[ApiRoute(verb: 'GET', url: '/api/2/{nodeCollection}/{nodeId}/rows/{rowId}', requirements: ['nodeCollection' => '(tables|views)', 'nodeId' => '(\d+)'])]
public function getRow(string $nodeCollection, int $nodeId, string $rowId): DataResponse {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure this addition belongs to this PR, what is the motivation?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3093 asked for it: the route section says "this needs a GET for both uuid and integer id, as this route does not exist yet", and one of the acceptance criteria covers GET, PUT and DELETE on that path. It was also where most of the extra logic came from: API v2 has no single-row read yet, so reading through a view needed its own permission, filter and column handling, including the zero-column changes you asked about below. Given your preference to keep UUIDs out of the core logic, I've removed it together with the UUID addressing, so this PR doesn't cover that acceptance criterion.

Comment thread lib/Controller/RowOCSController.php Outdated
#[RequirePermission(permission: Application::PERMISSION_UPDATE, typeParam: 'nodeCollection')]
#[ApiRoute(verb: 'PUT', url: '/api/2/{nodeCollection}/{nodeId}/rows/{rowId}', requirements: ['nodeCollection' => '(tables|views)', 'nodeId' => '(\d+)'])]
public function updateRow(string $nodeCollection, int $nodeId, int $rowId, mixed $data): DataResponse {
public function updateRow(string $nodeCollection, int $nodeId, string $rowId, mixed $data): DataResponse {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you consider creating a new endpoint to retrieve a row by UUID, and what let you to this decision?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I followed the design in #3093, which puts the UUID into the existing {rowId} routes and notes that rowId switches from int to string in their specification. The issue considered a separate /api/2/rows/{uuid} and rejected it, because API v2 already puts the table or view in the route and a UUID alone would leave the permission context ambiguous. I didn't evaluate a separate route under the node path. I've now taken UUID addressing out of this PR, so PUT and DELETE are unchanged from main.

Comment thread lib/Controller/RowOCSController.php Outdated
#[RequirePermission(permission: Application::PERMISSION_DELETE, typeParam: 'nodeCollection')]
#[ApiRoute(verb: 'DELETE', url: '/api/2/{nodeCollection}/{nodeId}/rows/{rowId}', requirements: ['nodeCollection' => '(tables|views)', 'nodeId' => '(\d+)'])]
public function deleteRow(string $nodeCollection, int $nodeId, int $rowId): DataResponse {
public function deleteRow(string $nodeCollection, int $nodeId, string $rowId): DataResponse {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above.

Comment thread lib/Db/Row2Mapper.php Outdated
*/
private function getRows(array $rowIds, array $columnIds): array {
if (empty($rowIds) || empty($columnIds)) {
if (empty($rowIds)) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's behind this change?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It let the new single-row GET return a row of a table or view without columns (just the sleeve). Since Row2Mapper::find() goes through getRows(), it affected the other single-row callers too, and the findAll() guard kept listings as they were. Both are gone now.

Comment thread lib/Db/Row2Mapper.php Outdated
$cellMapperCache = [];

while ($rowData = $result->fetchAssociative()) {
while ($result !== null && ($rowData = $result->fetchAssociative())) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't be necessary either?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same reason, removed as well.

Comment thread lib/Model/RowDataInput.php Outdated
reset($this->data);
}

public static function createFromInputArray(array $cells): self {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This addition also does not really belong into this PR, does it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it isn't about UUIDs. It checked each cell of the PUT body at the controller boundary (required keys, integer column IDs), so malformed cells got a 400. Removed.

Comment thread lib/Service/ImportService.php Outdated
try {
$this->rowService->updateSet($rowId, $this->viewId, $data, $this->userId, $this->tableId);
$this->countUpdatedRows++;
} catch (NotFoundError $e) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a behavorial change and also outside of the scope

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It followed from the error changes in RowService: upsertRow() counts an InternalError as one failed line but rethrows a NotFoundError, which ends the import. With those changes, a spreadsheet ID that matches no row would have stopped the whole import. Removed together with them.

Comment thread lib/Service/RowService.php Outdated
$e = new \Exception('Update row is not allowed.');
$this->logger->error($e->getMessage(), ['exception' => $e]);
throw new InternalError(static::class . ' - ' . __FUNCTION__ . ': ' . $e->getMessage());
throw new NotFoundError('Row not found in the requested view.');

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here and later one, the changes errors don't seem fitting to the PR either and are a behavioral change (not judging them).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, they're out, along with the other incidental changes in RowService. Since PUT and DELETE resolved the row before calling the service, I had switched the later checks to NotFoundError so they gave 404 as well. #3178 covers that change on its own.

foolfitz and others added 4 commits November 20, 2026 23:01
Add a nullable, uniquely indexed uuid column to tables_row_sleeves.
Every sleeve insertion, including legacy transfers with explicit IDs,
assigns a UUIDv7 before writing. An assigned identity is normalized to
lowercase and cannot change or be cleared. Stored values load unchanged,
so rows created before this release keep NULL until they are backfilled.

User migration archives now carry each row's UUID. Import keeps an
archived UUID unless another row already has it, in which case the
imported row gets a fresh one. Archives without a UUID, or with one that
is not a valid UUIDv7, get a fresh one as well, as table imports treat
invalid table UUIDs. The UUID is looked up before inserting, as
TableService::create() does for table UUIDs, because the import runs in
one transaction that a unique constraint violation would cancel.

Refs: nextcloud#3093
Assisted-by: Codex:gpt-6
Assisted-by: Codex:gpt-6-astra
Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-fable-5-2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jiajun Xu <foolfitz@gmail.com>
Register a periodic job from the migration that assigns UUIDv7 values
to sleeves still holding NULL. A run processes batches of up to 500
rows for about five seconds and commits each batch. The update only
touches rows whose uuid is still NULL, so concurrent assignments and
deletions are respected and audit metadata stays unchanged. Parallel
runs are disabled.

Progress is a lazy app-config cursor written once per run. A lagging
cursor only repeats a cheap NULL scan, while writing it after every
batch would invalidate the shared app-config cache. When a pass finds
nothing left, the job rescans from the start for rows below the cursor
before it removes itself. On SQLite the job pauses briefly between
batches so requests can take the single writer.

Refs: nextcloud#3093
Assisted-by: Codex:gpt-6
Assisted-by: Codex:gpt-6-astra
Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-fable-5-2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jiajun Xu <foolfitz@gmail.com>
Row responses now include the persisted uuid next to the integer id,
including public-share rows and the row returned by POST. Rows created
before the backfill reaches them report null. Row responses forwarded
from a federated server keep the remote uuid, or report null when the
remote server does not send one.

Regenerate openapi.json and the TypeScript types for the new field.

Refs: nextcloud#3093
Assisted-by: Codex:gpt-6
Assisted-by: Codex:gpt-6-astra
Assisted-by: ClaudeCode:claude-opus-5-5
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jiajun Xu <foolfitz@gmail.com>
Existing installations run the row UUID migration only when the declared
app version is higher than the installed one. Bump appinfo/info.xml along
with package.json and package-lock.json, which this repository keeps in
sync.

Refs: nextcloud#3093
Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-fable-5-2
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Jiajun Xu <foolfitz@gmail.com>
@foolfitz
foolfitz force-pushed the feat/3093-row-uuid-squashed branch from 266c10c to 8ab689f Compare November 20, 2026 15:01
@foolfitz

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I've cut the PR down to the row identity itself: the UUID column with its unique index, assignment on insert, the background backfill, keeping UUIDs in user migration archives, and uuid in row responses.

Addressing rows by UUID is no longer part of it. The API v2 row routes are back to main's numeric IDs (only federated responses still fill in uuid: null), and RowService::updateSet(), delete() and the row lookups in Row2Mapper are unchanged from main, so the extra work on PUT and DELETE is gone, together with the unrelated changes you pointed out. Archive import now checks for a used UUID before inserting, the same way TableService::create() does for table UUIDs, and the backfill job no longer retries collisions itself.

To get there I dropped the last two commits and rebased the rest onto current main, which also resolves the conflicts. The PR is now the first three commits, with the archive import change folded into the first, plus the version bump. I tested the upgrade from 2.4.0, the backfill, user migration imports and the unit suite on MySQL, MariaDB, PostgreSQL and SQLite.

@github-actions

Copy link
Copy Markdown
Contributor

Hello there,
Thank you so much for taking the time and effort to create a pull request to our Nextcloud project.

We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process.

Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6

Thank you for contributing to Nextcloud and we hope to hear from you soon!

(If you believe you should not receive this message, you can add yourself to the blocklist.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants