Repository navigation
fix(contexts): map each node of an imported application to its own table or view - #3200
rubenvdlinde wants to merge 7 commits into
Conversation
6d3a321 to
4603c99
Compare
2471ff4 to
6368c26
Compare
4603c99 to
88e3e69
Compare
6368c26 to
c203b3a
Compare
88e3e69 to
68dd2ab
Compare
c203b3a to
5cf92cf
Compare
68dd2ab to
35d8b9e
Compare
AndyScherzinger
left a comment
There was a problem hiding this comment.
Review focused on performance, clean code and architecture. The checks ran on a dev instance (MariaDB) with this PR on top of #3138, exporting and importing accounts with occ user:export / occ user:import.
The main change holds: an application with two tables and a view now comes in once, with its relations on the new tables and view, its start page contents and widgets on those relations, and its menu items on the right targets.
Commits: 2101205 doesn't pass its own tests: it calls WidgetService::insert(), which doesn't exist in its tree (only insertAll()), and asserts [501, 503] where its code yields [501, 503, 501, 502, 503]; 35d8b9e repairs that and could be squashed into it, so git bisect over the stack doesn't stop on a red commit.
| $imported, | ||
| $context['menu'], | ||
| static fn (string $kind, string $uuid): ?int => $kind === 'table' && $uuid === $tableOldUuid && $tableRelation !== false ? $tableRelation->getId() : null, | ||
| fn (string $kind, string $uuid): ?int => $this->nodeRelIdByUuid($imported, $kind, $uuid), |
There was a problem hiding this comment.
Migrated menu items now resolve through nodeRelIdByUuid()'s instance-wide findByUuid(), and view uuids are unique per table only. A view uuid held by two views throws an uncaught MultipleObjectsReturnedException and rolls back the user's whole Tables import. Two tables created from the same scheme are enough (createFromScheme gives the table a new uuid but keeps the view uuid), and so is any other user owning a view with that uuid, so one user can block another's migration. Each view item also costs a query that scans tables_tables. Resolving uuids among the application's own nodes (findMany on its node ids, as MenuService::export() does), or via uuid → old id → the id maps from tables.json/views.json, fixes it and removes the query per item. The tests mock findByUuid to return one row.
Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 35d8b9ef, archives from occ user:export -t tables imported with occ user:import: two tables from the same scheme with a menu item on one of the views; the import printed "Did not expect more than one result when executing: query SELECT v.*, t.ownership FROM tables_views ..." and imported nothing; the same archive imports with #3138).
There was a problem hiding this comment.
Agreed, an instance-wide uuid lookup was the wrong tool here. The migrator now hands the import the ids of the tables and views it created, by exported uuid, and a menu item takes the one that is a node of the application, so two views sharing a uuid cannot break it and there is no query per item. Tests cover two views sharing a uuid, in the unit test and against the database.
cc @blizzz
| } | ||
| $view = $this->remapViewCardSources($view, $columnIdMap); | ||
| $this->viewService->importView($newTableId, $view, $userId); | ||
| $viewIdMap[$view['id']] = $this->viewService->importView($newTableId, $view, $userId)->getId(); |
There was a problem hiding this comment.
importViews() remaps column settings and card sources but not filter and sort, so every migrated view keeps the source's column ids. This is older, but this PR now points application view nodes, data widgets and menu items at these views: the view returns no rows, and on a target where the old column id doesn't exist Row2Mapper::getFilter() throws "No column found to build filter with for id ...", so the view and its widgets don't load. createFromScheme already remaps both through $colMap.
Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 35d8b9ef, archives from occ user:export -t tables imported with occ user:import: a view filtered "Status is-equal open" on column 19 came back with filter column 19 while its own column is 20, and returned [] instead of the open row).
There was a problem hiding this comment.
Good catch, and it matters more now that pages, widgets and menu items point at these views. The migrator now remaps the column of every filter rule and sort rule through the column id map, as it already did for column settings; meta columns keep their ids. A test covers a filtered and sorted view.
cc @blizzz
| } | ||
|
|
||
| if (!$hasMatchingNode) { | ||
| if ($nodes === []) { |
There was a problem hiding this comment.
The import gate is "some node was imported", not ownership. An application the user owns without tables or views (grid pages of text and header widgets, a menu of links, which this stack makes meaningful) is dropped without a log line, and an application only shared with the user becomes the user's own as soon as one of its nodes maps, now also through view nodes. The archive carries owner and IImportSource::getOriginalUid() gives the exporting user, so the gate can be ownership; dropped nodes should be logged too.
Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 35d8b9ef, archives from occ user:export -t tables imported with occ user:import: an own application with no nodes, a text-widget page and a link menu item; the import said "Import completed" but no application was created and nothing was logged).
There was a problem hiding this comment.
You are right that node count was a poor stand-in for ownership. The migrator now imports the applications whose exported owner is the user the data came from, also those with only text pages or links, and skips the others with a line in the import output. A node whose table or view did not come along is left out with a log line.
cc @blizzz
|
|
||
| if (is_array($context['menu'] ?? null)) { | ||
| $tableRelation = reset($sourceRelMap); | ||
| $imported = $this->contextMapper->findById($newContext->getId()); |
There was a problem hiding this comment.
Every application without menu items still pays a full findById() reload and MenuService::import()'s nested transaction with a DELETE, because the export always writes menu: []: 2 SELECTs, a SAVEPOINT, a DELETE and a RELEASE per application inside the migration transaction, for nothing. A $context['menu'] !== [] guard avoids it; with a node-based uuid map (comment on line 766) the reload isn't needed for non-empty menus either.
Validated: code check only.
There was a problem hiding this comment.
Agreed. An empty menu now skips the menu import, and since items resolve through the id maps the application is no longer reloaded for a non-empty menu either.
cc @blizzz
| if ($oldRelId === null) { | ||
| continue; | ||
| } | ||
| foreach ($nodes as [$nodeArr, $nodeType, $nodeId]) { |
There was a problem hiding this comment.
The rewritten node loop re-implements addNodeToContext() line by line, so relations are now created in two places and later changes to addNodeToContext() miss imports. It also carries the exported node in a positional [$nodeArr, type, id] tuple only to read permissions and id again, and the archive's permissions go into the entity setter unchecked. Returning a named shape {oldRelId, nodeType, nodeId, permissions} from importedNode() (permissions as int or 0) and calling addNodeToContext() keeps one path. The docblock of importContext() should also state that the id maps must hold only ids the import just created, and document $ownerId, the return value and the menu handling; importContexts() misses $userId.
Validated: code check only.
There was a problem hiding this comment.
Fair point, two places creating relations would drift. The import now calls addNodeToContext() with a named shape from importedNode(), and permissions that are not a number become 0. I also completed the docblocks of importContext() and importContexts().
cc @blizzz
|
|
||
| public function testAnApplicationOfTwoTablesAndAViewIsImportedWithEachNodeOnItsOwnTarget(): void { | ||
| $this->permissionsService->method('canManageNodeById')->willReturn(true); | ||
| $this->tableMapper->method('findMany')->willReturnCallback(static fn (array $ids): array => array_combine($ids, array_map(static fn (): Table => new Table(), $ids))); |
There was a problem hiding this comment.
Small test cleanups: the new round-trip test stubs TableMapper::findMany(), which ContextService never calls here, and promotes two mocks to typed properties for it; it adds writes to $contextIds, which tearDown() never reads; MenuServiceTest builds the relation map with array_flip(array_column(..., 'node_id', 'id')) twice where array_column($nodes, 'id', 'node_id') does it; TablesMigratorTest adds a sixth copy of the anonymous connection class where createMock(IDBConnection::class) works. No test has two views sharing a uuid, which is why the comment on ContextService.php line 766 passes CI.
Validated: code check only.
There was a problem hiding this comment.
Thanks, all fair. I removed the unused findMany stub and its mock, the unread contextIds, simplified the relation map to array_column(..., 'id', 'node_id'), used a plain connection mock in the migrator test I added, and added tests with two views sharing a uuid.
cc @blizzz
5cf92cf to
ee0564c
Compare
35d8b9e to
5ca9827
Compare
ee0564c to
00c1b92
Compare
5ca9827 to
4b8594c
Compare
00c1b92 to
6ffaf94
Compare
4b8594c to
768fff0
Compare
|
Hello there, We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process. Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6 Thank you for contributing to Nextcloud and we hope to hear from you soon! (If you believe you should not receive this message, you can add yourself to the blocklist.) |
6ffaf94 to
f7549c7
Compare
768fff0 to
a59c7b3
Compare
f7549c7 to
b464a42
Compare
a59c7b3 to
13da89d
Compare
b464a42 to
c0627d7
Compare
13da89d to
a050f63
Compare
|
You are right, the first commit was red on its own. I folded the test fix into it, so it now passes its own tests, and every commit I added after it does too. cc @blizzz |
…ble or view Moving a user's data imported an application once for every table it held, and pointed every node of each copy at that one table, views included. The migrator now imports the tables and views first and then each application once, mapping every node to the table or view it was imported as. Page contents and widget sources follow the node they showed, and menu items that open a table or a view keep theirs; a node that was not imported is left out. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…own nodes A menu item of a moved user's application looked up its table or view by uuid across the whole instance, which fails the import when two views share a uuid and costs a query per item. The migrator now hands the import the ids of the tables and views it created by exported uuid, and an item takes the one that is a node of the application. An empty menu is not imported at all, and the application is no longer reloaded for it. Imported nodes go through addNodeToContext() with a named shape; permissions that are not a number become none. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…thout nodes The import took an application when one of its nodes came along, so an own application holding only text pages and links was dropped without a word, and an application shared with the user became theirs once one of its tables mapped. The migrator now imports the applications whose exported owner is the user the data was exported from, and skips the others with a line in the output. A node whose table or view was not imported is left out with a log line. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
… columns A moved user's views kept the column ids of their filter and sort rules from the source instance, so a filtered view showed no rows or failed, while its column settings were already remapped. The migrator now remaps the column of every filter rule and sort rule through the column id map; meta columns keep their ids. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…ests Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…son of the node map A menu export now reads the titles of its tables and views from the entities it loads, so the view of the import tests gets a title, and the two tests that open it get the same stubs. An item for a table that did not come along is skipped as not part of the application, as the import now finds every table and view among the nodes it created. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…ne of its parameter Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
c0627d7 to
b57a52f
Compare
a050f63 to
c828682
Compare
Part of #3099, on top of #3138 (menu export and import). When a user's data is moved to another instance (
TablesMigrator), an application now comes along once, with every table and view in it pointing at the table or view it was imported as.Size: 364 lines: 133 code and 231 tests.
What was wrong
TablesMigrator::import()calledContextService::importContext()for every application once per imported table, before the views were imported.importContext()then pointed every node of the application at that one table (setNodeId($table->getId())), whatever the node was. An application with two tables and a view therefore came out as two applications, each with three nodes on the same table, the view node included. Measured with that application on an earlier version of this stack, which had the same import:What changes
importViews()now returns the view ids), and then imports each application once.importContext(array $context, string $ownerId, array $tableIds, array $viewIds)maps every node to its own imported table or view. A node whose table or view didn't come along is left out, and so are the page contents and widget sources that showed it. An application none of whose nodes came along is not imported, as before.How it was tested
On Nextcloud 34 with PostgreSQL:
ContextPagesRoundTripTest: a database round trip of an application with two tables, a view and a widget on each. Every node, start page content and widget source lands on its own table or view.MenuServiceTest: a menu with an item on a table and an item on a view is exported and imported with new node ids. Both items open the new relation of their node, and an item on a view that didn't come along is skipped. With views left unmapped, this test, the round trip above and theContextServiceTestone below fail.ContextServiceTest: nodes, page contents and widget sources of tables, a view and a node that didn't come along, the owner, and an application without imported nodes.TablesMigratorTest: an application with two tables and a view is imported once, with the ids of both tables and the view.composer cs:check, psalm and rector, all cleanplaywright/e2e/context.spec.tson PostgreSQL with Nextcloud 34: 7 passed. Its first test, "Update and add resources", also passed on SQLite with Nextcloud 35The user migration itself (
occ user:export/user:import) was not run: theuser_migrationapp is not on the test instance.🖼️ Screenshots
No UI changes.
🏁 Checklist
/backport to stableX.X🤖 AI (if applicable)