Skip to content

fix(contexts): map each node of an imported application to its own table or view - #3200

Open
rubenvdlinde wants to merge 7 commits into
feat/noid/app-menu-export-importfrom
fix/noid/import-context-node-mapping
Open

rubenvdlinde wants to merge 7 commits into
feat/noid/app-menu-export-importfrom
fix/noid/import-context-node-mapping

Conversation

@rubenvdlinde

@rubenvdlinde rubenvdlinde commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Part of #3099, on top of #3138 (menu export and import). When a user's data is moved to another instance (TablesMigrator), an application now comes along once, with every table and view in it pointing at the table or view it was imported as.

Size: 364 lines: 133 code and 231 tests.

What was wrong

TablesMigrator::import() called ContextService::importContext() for every application once per imported table, before the views were imported. importContext() then pointed every node of the application at that one table (setNodeId($table->getId())), whatever the node was. An application with two tables and a view therefore came out as two applications, each with three nodes on the same table, the view node included. Measured with that application on an earlier version of this stack, which had the same import:

imports: 2 relations: [[0,5151],[0,5151],[1,5151],[0,5152],[0,5152],[1,5152]]

What changes

  • The migrator imports the tables and views first, keeps the old→new ids of both (importViews() now returns the view ids), and then imports each application once.
  • importContext(array $context, string $ownerId, array $tableIds, array $viewIds) maps every node to its own imported table or view. A node whose table or view didn't come along is left out, and so are the page contents and widget sources that showed it. An application none of whose nodes came along is not imported, as before.
  • Widget sources follow their node for every mapped node. Before, a source was only kept when it showed the table being imported, because every other node pointed at the wrong table.
  • Menu items that open a table or a view keep it. The import finds them by the uuid in the export, the way the scheme import does, among the nodes of the application. The migrator imports tables and views with the uuid they had, so this lifts the skipping of view items as well. An item whose table or view didn't come along is skipped with the reason in the log.

How it was tested

On Nextcloud 34 with PostgreSQL:

  • ContextPagesRoundTripTest: a database round trip of an application with two tables, a view and a widget on each. Every node, start page content and widget source lands on its own table or view.
  • MenuServiceTest: a menu with an item on a table and an item on a view is exported and imported with new node ids. Both items open the new relation of their node, and an item on a view that didn't come along is skipped. With views left unmapped, this test, the round trip above and the ContextServiceTest one below fail.
  • ContextServiceTest: nodes, page contents and widget sources of tables, a view and a node that didn't come along, the owner, and an application without imported nodes.
  • TablesMigratorTest: an application with two tables and a view is imported once, with the ids of both tables and the view.
  • the full unit suite on PostgreSQL: 617 tests
  • composer cs:check, psalm and rector, all clean
  • playwright/e2e/context.spec.ts on PostgreSQL with Nextcloud 34: 7 passed. Its first test, "Update and add resources", also passed on SQLite with Nextcloud 35

The user migration itself (occ user:export / user:import) was not run: the user_migration app is not on the test instance.

🖼️ Screenshots

No UI changes.

🏁 Checklist

  • ⛑️ Tests (unit and/or integration) are included or not needed
  • 🔙 Backport requests are created or not needed: /backport to stableX.X
  • 📅 Milestone is set
  • 🌸 PR title is meaningful (if it should be in the changelog: is it meaningful to users?)

🤖 AI (if applicable)

  • The content of this PR was partly or fully generated using AI

@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from 6d3a321 to 4603c99 Compare November 3, 2026 14:51
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/app-menu-export-import branch from 2471ff4 to 6368c26 Compare November 4, 2026 12:35
@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from 4603c99 to 88e3e69 Compare November 4, 2026 12:35
@rubenvdlinde
rubenvdlinde marked this pull request as ready for review November 6, 2026 08:24
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/app-menu-export-import branch from 6368c26 to c203b3a Compare November 10, 2026 09:29
@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from 88e3e69 to 68dd2ab Compare November 10, 2026 09:29
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/app-menu-export-import branch from c203b3a to 5cf92cf Compare November 10, 2026 10:13
@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from 68dd2ab to 35d8b9e Compare November 10, 2026 10:13

@AndyScherzinger AndyScherzinger left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review focused on performance, clean code and architecture. The checks ran on a dev instance (MariaDB) with this PR on top of #3138, exporting and importing accounts with occ user:export / occ user:import.

The main change holds: an application with two tables and a view now comes in once, with its relations on the new tables and view, its start page contents and widgets on those relations, and its menu items on the right targets.

Commits: 2101205 doesn't pass its own tests: it calls WidgetService::insert(), which doesn't exist in its tree (only insertAll()), and asserts [501, 503] where its code yields [501, 503, 501, 502, 503]; 35d8b9e repairs that and could be squashed into it, so git bisect over the stack doesn't stop on a red commit.

Comment thread lib/Service/ContextService.php Outdated
$imported,
$context['menu'],
static fn (string $kind, string $uuid): ?int => $kind === 'table' && $uuid === $tableOldUuid && $tableRelation !== false ? $tableRelation->getId() : null,
fn (string $kind, string $uuid): ?int => $this->nodeRelIdByUuid($imported, $kind, $uuid),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Migrated menu items now resolve through nodeRelIdByUuid()'s instance-wide findByUuid(), and view uuids are unique per table only. A view uuid held by two views throws an uncaught MultipleObjectsReturnedException and rolls back the user's whole Tables import. Two tables created from the same scheme are enough (createFromScheme gives the table a new uuid but keeps the view uuid), and so is any other user owning a view with that uuid, so one user can block another's migration. Each view item also costs a query that scans tables_tables. Resolving uuids among the application's own nodes (findMany on its node ids, as MenuService::export() does), or via uuid → old id → the id maps from tables.json/views.json, fixes it and removes the query per item. The tests mock findByUuid to return one row.

Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 35d8b9ef, archives from occ user:export -t tables imported with occ user:import: two tables from the same scheme with a menu item on one of the views; the import printed "Did not expect more than one result when executing: query SELECT v.*, t.ownership FROM tables_views ..." and imported nothing; the same archive imports with #3138).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, an instance-wide uuid lookup was the wrong tool here. The migrator now hands the import the ids of the tables and views it created, by exported uuid, and a menu item takes the one that is a node of the application, so two views sharing a uuid cannot break it and there is no query per item. Tests cover two views sharing a uuid, in the unit test and against the database.

cc @blizzz

}
$view = $this->remapViewCardSources($view, $columnIdMap);
$this->viewService->importView($newTableId, $view, $userId);
$viewIdMap[$view['id']] = $this->viewService->importView($newTableId, $view, $userId)->getId();

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

importViews() remaps column settings and card sources but not filter and sort, so every migrated view keeps the source's column ids. This is older, but this PR now points application view nodes, data widgets and menu items at these views: the view returns no rows, and on a target where the old column id doesn't exist Row2Mapper::getFilter() throws "No column found to build filter with for id ...", so the view and its widgets don't load. createFromScheme already remaps both through $colMap.

Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 35d8b9ef, archives from occ user:export -t tables imported with occ user:import: a view filtered "Status is-equal open" on column 19 came back with filter column 19 while its own column is 20, and returned [] instead of the open row).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, and it matters more now that pages, widgets and menu items point at these views. The migrator now remaps the column of every filter rule and sort rule through the column id map, as it already did for column settings; meta columns keep their ids. A test covers a filtered and sorted view.

cc @blizzz

Comment thread lib/Service/ContextService.php Outdated
}

if (!$hasMatchingNode) {
if ($nodes === []) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The import gate is "some node was imported", not ownership. An application the user owns without tables or views (grid pages of text and header widgets, a menu of links, which this stack makes meaningful) is dropped without a log line, and an application only shared with the user becomes the user's own as soon as one of its nodes maps, now also through view nodes. The archive carries owner and IImportSource::getOriginalUid() gives the exporting user, so the gate can be ownership; dropped nodes should be logged too.

Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 35d8b9ef, archives from occ user:export -t tables imported with occ user:import: an own application with no nodes, a text-widget page and a link menu item; the import said "Import completed" but no application was created and nothing was logged).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right that node count was a poor stand-in for ownership. The migrator now imports the applications whose exported owner is the user the data came from, also those with only text pages or links, and skips the others with a line in the import output. A node whose table or view did not come along is left out with a log line.

cc @blizzz

Comment thread lib/Service/ContextService.php Outdated

if (is_array($context['menu'] ?? null)) {
$tableRelation = reset($sourceRelMap);
$imported = $this->contextMapper->findById($newContext->getId());

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Every application without menu items still pays a full findById() reload and MenuService::import()'s nested transaction with a DELETE, because the export always writes menu: []: 2 SELECTs, a SAVEPOINT, a DELETE and a RELEASE per application inside the migration transaction, for nothing. A $context['menu'] !== [] guard avoids it; with a node-based uuid map (comment on line 766) the reload isn't needed for non-empty menus either.

Validated: code check only.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. An empty menu now skips the menu import, and since items resolve through the id maps the application is no longer reloaded for a non-empty menu either.

cc @blizzz

Comment thread lib/Service/ContextService.php Outdated
if ($oldRelId === null) {
continue;
}
foreach ($nodes as [$nodeArr, $nodeType, $nodeId]) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rewritten node loop re-implements addNodeToContext() line by line, so relations are now created in two places and later changes to addNodeToContext() miss imports. It also carries the exported node in a positional [$nodeArr, type, id] tuple only to read permissions and id again, and the archive's permissions go into the entity setter unchecked. Returning a named shape {oldRelId, nodeType, nodeId, permissions} from importedNode() (permissions as int or 0) and calling addNodeToContext() keeps one path. The docblock of importContext() should also state that the id maps must hold only ids the import just created, and document $ownerId, the return value and the menu handling; importContexts() misses $userId.

Validated: code check only.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point, two places creating relations would drift. The import now calls addNodeToContext() with a named shape from importedNode(), and permissions that are not a number become 0. I also completed the docblocks of importContext() and importContexts().

cc @blizzz


public function testAnApplicationOfTwoTablesAndAViewIsImportedWithEachNodeOnItsOwnTarget(): void {
$this->permissionsService->method('canManageNodeById')->willReturn(true);
$this->tableMapper->method('findMany')->willReturnCallback(static fn (array $ids): array => array_combine($ids, array_map(static fn (): Table => new Table(), $ids)));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small test cleanups: the new round-trip test stubs TableMapper::findMany(), which ContextService never calls here, and promotes two mocks to typed properties for it; it adds writes to $contextIds, which tearDown() never reads; MenuServiceTest builds the relation map with array_flip(array_column(..., 'node_id', 'id')) twice where array_column($nodes, 'id', 'node_id') does it; TablesMigratorTest adds a sixth copy of the anonymous connection class where createMock(IDBConnection::class) works. No test has two views sharing a uuid, which is why the comment on ContextService.php line 766 passes CI.

Validated: code check only.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, all fair. I removed the unused findMany stub and its mock, the unread contextIds, simplified the relation map to array_column(..., 'id', 'node_id'), used a plain connection mock in the migrator test I added, and added tests with two views sharing a uuid.

cc @blizzz

@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/app-menu-export-import branch from 5cf92cf to ee0564c Compare November 13, 2026 21:04
@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from 35d8b9e to 5ca9827 Compare November 13, 2026 21:04
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/app-menu-export-import branch from ee0564c to 00c1b92 Compare November 16, 2026 09:23
@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from 5ca9827 to 4b8594c Compare November 16, 2026 09:23
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/app-menu-export-import branch from 00c1b92 to 6ffaf94 Compare November 17, 2026 09:11
@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from 4b8594c to 768fff0 Compare November 17, 2026 09:11
@github-actions

Copy link
Copy Markdown
Contributor

Hello there,
Thank you so much for taking the time and effort to create a pull request to our Nextcloud project.

We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process.

Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6

Thank you for contributing to Nextcloud and we hope to hear from you soon!

(If you believe you should not receive this message, you can add yourself to the blocklist.)

@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/app-menu-export-import branch from 6ffaf94 to f7549c7 Compare November 19, 2026 08:28
@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from 768fff0 to a59c7b3 Compare November 19, 2026 08:28
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/app-menu-export-import branch from f7549c7 to b464a42 Compare November 19, 2026 13:36
@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from a59c7b3 to 13da89d Compare November 19, 2026 13:36
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/app-menu-export-import branch from b464a42 to c0627d7 Compare November 20, 2026 10:53
@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from 13da89d to a050f63 Compare November 20, 2026 10:53
@rubenvdlinde

Copy link
Copy Markdown
Contributor Author

You are right, the first commit was red on its own. I folded the test fix into it, so it now passes its own tests, and every commit I added after it does too.

cc @blizzz

…ble or view

Moving a user's data imported an application once for every table it
held, and pointed every node of each copy at that one table, views
included. The migrator now imports the tables and views first and then
each application once, mapping every node to the table or view it was
imported as. Page contents and widget sources follow the node they
showed, and menu items that open a table or a view keep theirs; a node
that was not imported is left out.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…own nodes

A menu item of a moved user's application looked up its table or view
by uuid across the whole instance, which fails the import when two
views share a uuid and costs a query per item. The migrator now hands
the import the ids of the tables and views it created by exported
uuid, and an item takes the one that is a node of the application. An
empty menu is not imported at all, and the application is no longer
reloaded for it.

Imported nodes go through addNodeToContext() with a named shape;
permissions that are not a number become none.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…thout nodes

The import took an application when one of its nodes came along, so an
own application holding only text pages and links was dropped without
a word, and an application shared with the user became theirs once one
of its tables mapped. The migrator now imports the applications whose
exported owner is the user the data was exported from, and skips the
others with a line in the output. A node whose table or view was not
imported is left out with a log line.

Assisted-by: ClaudeCode:claude-opus-5-5

Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
… columns

A moved user's views kept the column ids of their filter and sort
rules from the source instance, so a filtered view showed no rows or
failed, while its column settings were already remapped. The migrator
now remaps the column of every filter rule and sort rule through the
column id map; meta columns keep their ids.

Assisted-by: ClaudeCode:claude-opus-5-5

Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…ests

Assisted-by: ClaudeCode:claude-opus-5-5

Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…son of the node map

A menu export now reads the titles of its tables and views from the
entities it loads, so the view of the import tests gets a title, and the
two tests that open it get the same stubs. An item for a table that did
not come along is skipped as not part of the application, as the import
now finds every table and view among the nodes it created.

Assisted-by: ClaudeCode:claude-opus-5-5

Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…ne of its parameter

Assisted-by: ClaudeCode:claude-opus-5-5

Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/app-menu-export-import branch from c0627d7 to b57a52f Compare November 21, 2026 10:53
@rubenvdlinde
rubenvdlinde force-pushed the fix/noid/import-context-node-mapping branch from a050f63 to c828682 Compare November 21, 2026 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants