Repository navigation
feat(contexts): carry pages and their widgets through scheme export and import - #3201
rubenvdlinde wants to merge 13 commits into
Conversation
c30848c to
4010828
Compare
2471ff4 to
6368c26
Compare
4010828 to
8da4db7
Compare
6368c26 to
c203b3a
Compare
8da4db7 to
0d1a23c
Compare
c203b3a to
5cf92cf
Compare
0d1a23c to
30de022
Compare
AndyScherzinger
left a comment
There was a problem hiding this comment.
Review focused on performance, accessibility, security, clean code and architecture. The checks ran on a dev instance (MariaDB) with this PR on top of #3138: scheme export, preview and import through the OCS API with the payload the dialog builds, and the import dialog in the browser.
Pages and their widgets do make the round trip. The main problem is how pages are matched (comment on line 1321): the automatic page_<id> names are row ids of the exporting server, so a scheme can overwrite an unrelated page here, and every re-import of the same scheme adds the automatically named pages again.
Accessibility: I checked the new Pages section of the import preview in the browser with axe-core 4.14 and the keyboard. The switch is named "Pages", toggles with Space, keeps the focus, and hides the lists when off. axe reports only heading-order on the existing h4 section headings, which this PR doesn't change. What the lists say is in the comments on ImportContextSchemePreview.vue lines 237 and 404.
CI: rector strict fails as on the base (inherited from #3133).
| continue; | ||
| } | ||
| $pageId = $startPageId; | ||
| } elseif ($name !== null && isset($pageIdsByName[$name])) { |
There was a problem hiding this comment.
resolveSchemePages() matches scheme pages to local pages by technical name, and the automatic names are page_<id>, a row id of the exporting server. Automatic names are the default (the page form says "Leave empty to get one automatically"). Line 1323 already treats page_<id> as not portable for new pages, but here it is trusted for matching. Across servers, the scheme's page_105 meets this server's unrelated grid page 105: its title, icon and description are replaced, and syncPage() deletes all its widgets. Nothing in the preview shows that a page will be overwritten. On the same server, the scheme's page_113 never matches the target's new page, so each re-import adds another copy, and the menu item follows only the newest one. Matching automatically named pages by something portable (their position, or a uuid as tables and views have), or treating them as new and never as a match, avoids both.
Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 30de0225, scheme preview and import through the OCS API as alice: the export of an application with grid page "Overview" (page_113) imported twice into another application gave it page_116 and page_117, both "Overview"; a scheme page named like an existing page_<id> of the target replaced that page's title and widgets).
There was a problem hiding this comment.
Pages now have an immutable slug, set when the page is added and carried by export and import. A scheme imported into an application updates the page with the same chosen slug and the same page type, and adds the others. An automatic page_<id> slug never matches, because it names a page of the server the scheme came from; such a page is always added as a new page. Once applications get their own slug from #3094, the match will be on the application slug and the page slug together, and automatic slugs can then match within the same application.
cc @blizzz
| $keptEntries = []; | ||
| foreach (is_array($pageArr['widgets'] ?? null) ? array_values($pageArr['widgets']) : [] as $widgetIndex => $widgetArr) { | ||
| if (!$isGrid) { | ||
| $skip('only grid pages hold widgets', $widgetIndex, $widgetArr); |
There was a problem hiding this comment.
Widgets on the scheme's start page are always skipped with "only grid pages hold widgets", but the page API stores widgets on start pages, the export carries them, and the account import keeps them (line 786). Every scheme round trip therefore loses the start page's widgets. #3211 (widgets on the start page) and #3216 (the start page shown as a grid page) make this the normal case.
Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 30de0225, scheme preview and import through the OCS API as alice: a header widget saved on the start page with PATCH (200), present in the export, listed in the preview into another application as skipped with "only grid pages hold widgets").
There was a problem hiding this comment.
You are right, the start page holds widgets through the API and the account import, so the scheme import keeps them too. The start page's widgets now go through the same path as a grid page's, and a round-trip test exports a start page with a header widget and imports it back.
cc @blizzz
| if ($name !== null) { | ||
| $usedNames[$name] = true; | ||
| } | ||
| $isGrid = $pageId === null || $pageId !== $startPageId; |
There was a problem hiding this comment.
Page matching ignores the page type. Start pages can be renamed (PageService::update() accepts technicalName on any page), so a source application with start page home and a grid page dashboard, imported into an application whose start page has the default name dashboard, resolves that grid page to the start page. Its title, icon and description overwrite the start page's, its widgets are skipped (comment on line 1335), and no grid page is created. MenuService::schemePageIds() binds that page's menu items to the start page too. Matching only pages of the same type fixes both.
Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 30de0225, scheme preview and import through the OCS API as alice: the scheme's grid page dashboard ("Grid page named dashboard", icon star, one text widget) turned the target's start page into "Grid page named dashboard" with icon star, and the widget was skipped).
There was a problem hiding this comment.
Good find. Pages match on slug and page type together: the start page of a scheme updates the start page, and a grid page only ever matches a grid page with its slug. A grid page named like the start page becomes a new grid page, and its menu item opens that new page; a test covers it. The menu takes its page matches from the same resolver, keyed by the slug in the scheme, so it cannot land elsewhere.
cc @blizzz
| $pageIds[$resolved['exportedId']] = $page->getId(); | ||
| } | ||
| if ($resolved['isGrid']) { | ||
| $this->widgetService->syncPage($page->getId(), array_map( |
There was a problem hiding this comment.
importSchemePages() writes page fields and replaces widgets outside PageService's save protocol. It never calls raiseVersion() (the version bump plus the page-row lock), and it never calls lockContext() before inserting pages. A save based on the version from before the import therefore silently undoes the import, although GridPage.vue sends the version so that "a save by someone else in between is not undone". When name, icon and description don't change, no context row is locked either, so the import isn't ordered against PageService::delete() or ContextService::delete() and can leave widgets or pages behind for a page or application deleted at the same time. Going through PageService (or calling raiseVersion() per matched page and lockContext() first) keeps one protocol.
Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 30de0225, scheme preview and import through the OCS API as alice: the page was at version 1 before and after an import that replaced its widgets; a save with version 1 and one new widget returned 200 and the imported widgets were gone). The delete races: code check only.
There was a problem hiding this comment.
The application lock comes first: the import runs update() before it writes pages, and that takes the application row lock in the same transaction, which a comment at the call now says. The version was missing. The import now raises the version of every page it updates before writing it, and a test checks that a save based on the version from before the import is refused.
cc @blizzz
| $addedUuids[$table['uuid']] = true; | ||
| } | ||
| foreach (is_array($table['views'] ?? null) ? $table['views'] : [] as $view) { | ||
| if (is_array($view) && is_string($view['uuid'] ?? null) && $this->findNodeByUuid('view', $view['uuid']) === null) { |
There was a problem hiding this comment.
addedSchemeUuids() and previewSchemeNodes() call findNodeByUuid() for every view of every scheme table, so the preview, which looked up no uuids before this PR, now also fails with the MultipleObjectsReturnedException posted on #3138. View uuids are unique per table only, and creating a table from another table's scheme (the template flow, createFromScheme()) copies them. After that, the preview of every application holding one of those tables answers 500, so the import dialog never gets past the preview, and every export carries pages, so this hits every such application.
Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 30de0225, scheme preview and import through the OCS API as alice: after two tables were created from table 1's scheme, the preview of an application's own export answered 500 (MultipleObjectsReturnedException from ContextService.php line 849, via lines 1255 and 1167); the same scheme without pages answered 200).
There was a problem hiding this comment.
Agreed, this was the same instance-wide uuid lookup as on #3138. The preview now looks a view up only within its own table among the tables of the scheme, plus the application's current nodes, so views that share a uuid no longer break it. A test makes the instance-wide lookup throw and the preview still resolves the view in the right table.
cc @blizzz
| */ | ||
| private function importSchemePages(int $contextId, array $pages, string $userId): array { | ||
| $relIds = []; | ||
| foreach ($this->contextMapper->findById($contextId, $userId)->getNodes() ?? [] as $relId => $node) { |
There was a problem hiding this comment.
An import with pages loads the application five times (ContextMapper::findById() at lines 1520, 197, 1397, 1528 and 1537; three are new), each a joined query plus a query for the other pages. Line 1520 only needs pages, and lines 1397 and 1528 read the same nodes. Each page then costs 2-3 avoidable statements (findById(), syncPage()'s closing re-read whose result is thrown away, and its FOR UPDATE on a page inserted a moment earlier), which insertAll(), used by importContext(), avoids, and each widget builds GridWidgetTypes::all() three times (36 l10n->t() calls and two file_exists() checks for its icon), in the preview and again in the import. Passing the loaded application and the type list down removes them.
Validated: code check only.
There was a problem hiding this comment.
Thanks for counting. The import now loads the application once after the update for both the pages and the menu, plus once for the answer, takes matched pages from that load instead of findById(), gives new pages their widgets through insertAll(), and the widget types are built once per request. A test counts the loads.
cc @blizzz
| } | ||
| $source = $widgetArr['source'] ?? null; | ||
| $widgetArr['source'] = null; | ||
| if (isset($widgetArr['icon']) && (!is_string($widgetArr['icon']) || !IconHelper::isValid($widgetArr['icon']))) { |
There was a problem hiding this comment.
readImportedWidget() re-implements importWidgets()' preprocessing instead of sharing it: the unknown-icon rule now exists three times (lines 951, 983 and importedIcon() at 1012), and the "no source on a type that shows none" rule in two forms. readWidget() also still rejects a non-integer widget id that the import throws away anyway: a scheme widget with id "12" is skipped with "The id of widget 0 must be a number.", although withId(null) drops the id. $widgetArr['icon'] = $this->importedIcon($widgetArr['icon'] ?? null), removing id before readWidget(), and one reader shared with importWidgets() keep the two in step.
Validated: code check only.
There was a problem hiding this comment.
You are right, the rules should live in one place. Both imports now prepare a widget through one helper that drops the id, clears an unknown icon and clears the source on types that show none, so a widget with id "12" imports fine.
cc @blizzz
| * takes the title, icon and description of the scheme, and the widgets of a grid page | ||
| * replace its widgets. A new page without a free technical name gets the default name. | ||
| * | ||
| * @param list<array{exportedId: ?int, pageId: ?int, technicalName: ?string, isGrid: bool, title: ?string, icon: ?string, description: ?string, widgets: list<array{widget: WidgetInput, source: ?array{0: int, 1: ?int}}>, ...<array-key, mixed>}> $pages |
There was a problem hiding this comment.
A few AGENTS.md rules: the shape ..., ...<array-key, mixed>}> $pages is open ("never leave a trailing ... in a shape literal"); ContextController.php lines 374-377 pass the raw ?array $pages into ContextService::importScheme() ("Never pass a raw unvalidated array into a service method"); lines 1307-1308 have an unreachable branch (|| !is_array($pageArr) and ?? 'it is not an object' never apply, because pageSkipReason() already returns a reason for every non-array); and ContextServiceTest.php line 58 keeps $this->logger = $logger = ... while line 75 passes $this->logger.
Validated: code check only.
There was a problem hiding this comment.
Thanks, all four are fixed: the shapes are closed, the controller refuses pages that are not objects with a 400 before it reaches the service, the dead branch is gone, and the test setup no longer keeps the unused local.
cc @blizzz
| <li class="preview-card__meta-title"> | ||
| {{ t('tables', 'To') }}: | ||
| </li> | ||
| <li v-for="(page, index) in previewData.pages.to" :key="index"> |
There was a problem hiding this comment.
The Pages section reuses the menu's From/To lists, which describe a replacement, but the page import merges: no page is removed. So From [three pages] / To [two pages] reads as "three pages go away", and an entry that overwrites an existing page (comment on ContextService.php line 1321, deleting its widgets) looks the same as one that adds a page. The start page is listed by its technical name ("dashboard") on both sides, widgets aren't listed at all (that "Overview" brings two widgets is visible nowhere), skipped entries go under "Skipped, not available here" without the reason the server sends ("a page holds at most 100 widgets", "only grid pages hold widgets", a validation error), untitled widgets all read "Page: Untitled widget" without type or table, and a blank title renders as an empty list item. pages.from also uses camelCase keys where pages.to uses snake_case. Marking each entry as added or updated, naming the start page, showing the widget count and the server's reason lets the manager see what will be saved.
Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 30de0225, scheme preview and import through the OCS API as alice, plus the dialog in Chromium: From "dashboard", To "dashboard", "Overview" for an application with a two-widget grid page).
There was a problem hiding this comment.
Agreed, From and To told the wrong story for a merge. The section now lists each page once, says whether it is new or which page it updates, names the start page, shows how many widgets come along and how many lose their table or view, and gives the server's reason for everything skipped. The preview has its own summary in one casing next to the pages it sends back, and from is gone.
cc @blizzz
| if (item.widgetIndex === null) { | ||
| return page | ||
| } | ||
| return t('tables', '{page}: {widget}', { page, widget: item.widget || t('tables', 'Untitled widget') }, undefined, { escape: false }) |
There was a problem hiding this comment.
skippedPageLabel() passes only {escape: false}, so @nextcloud/l10n still runs DOMPurify over the raw page and widget titles: the skipped list shows HTML entities or loses text, and screen readers read the entities too. Vue's {{ }} already escapes, so {escape: false, sanitize: false} (or plain interpolation) is enough.
Validated: reproduced on a deployed instance (MariaDB, S-3138 restored and this PR at 30de0225, scheme preview and import through the OCS API as alice, plus the dialog in Chromium: page "R&D overview" with skipped widget "Sales & Marketing " rendered as "R&D overview: Sales & Marketing", with "" gone).
There was a problem hiding this comment.
Thanks, that was a real bug. The labels now turn off sanitizing as well, since Vue already escapes the text, so titles with an ampersand or angle brackets show as typed. A component test covers "R&D overview" and "Sales & Marketing ".
cc @blizzz
5cf92cf to
ee0564c
Compare
30de022 to
1f623a4
Compare
ee0564c to
00c1b92
Compare
1f623a4 to
ce7c97a
Compare
00c1b92 to
6ffaf94
Compare
ce7c97a to
11ee0ae
Compare
|
Hello there, We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process. Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6 Thank you for contributing to Nextcloud and we hope to hear from you soon! (If you believe you should not receive this message, you can add yourself to the blocklist.) |
6ffaf94 to
f7549c7
Compare
11ee0ae to
59f8b95
Compare
f7549c7 to
b464a42
Compare
59f8b95 to
5dd2a84
Compare
b464a42 to
c0627d7
Compare
5dd2a84 to
d8c37f5
Compare
|
I ran rector on the files this PR changes. The only findings are RemoveExtraParametersRector hits on cc @blizzz |
…nd import A scheme now names the table or view of every widget source by targetType and targetUuid. Importing it with pages updates the start page and every grid page with the same technical name, adds the others, and replaces the widgets of each grid page. A widget is checked as the API would save it, and its table or view must be a node of the application once the nodes of the scheme are imported; a widget never adds a node. Skipped pages and widgets get a reason, in the log and in the preview, and the menu is read after the pages so it can open a page the same import adds. The import screen has a Pages switch next to the Menu switch. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
A scheme import reads its widgets with GridWidgetTypes::readWidget(), like the API and the account import, and keeps the uuid source aside for its own lookup. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
The scheme import now keeps the widgets of the start page, keeps a data widget whose table or view did not come along without its source, falls back to the technical name for a grid page without a title and raises the version of every page it updates. Pages are resolved once and the menu preview takes that result, views are looked up within their own table in one query, a scheme brings at most 50 pages, and a cut past a limit gives one skipped entry. Both imports prepare a widget through one helper, the application is loaded once after the update, and the controller refuses pages that are not objects. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
The Pages section lists each page once: new, or the page it updates, the start page by name, with its widget count, the widgets that lose their table or view, and the reason for everything skipped. Titles are interpolated without sanitizing, as Vue escapes them. With the Pages switch off, menu items for new pages move to skipped and are left out of the import. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
… for Psalm The import of the scheme pages declared a sealed shape of each resolved page, while it gets the pages as resolveSchemePages() returns them, with the entry, current title and unlinked widget count too. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…he application The scheme pages were matched to the pages of the application by technical name whatever their type, so a grid page named like the start page stood for the start page, and its menu item opened the start page a second time. A grid page of a scheme now only updates a grid page; one named like the start page is added with the default name. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
The pages of a scheme are matched to the pages of the application by slug and page type, and the menu of the scheme points at them by the same slug: the preview and the import take their page matches from one resolver keyed by the slug in the scheme. A grid page named like the start page becomes a new grid page and its menu item opens it. Without the pages, the dialog keeps the items of the pages that the scheme updates. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
A page_<id> slug names a page of the instance the scheme was exported from, so it never updates a page here; only a chosen slug of the same page type does. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
c0627d7 to
b57a52f
Compare
d8c37f5 to
367c70c
Compare
Part of #3099, on top of #3138 (menu export and import). An application's pages and their widgets now travel with its scheme, the same way the menu does: pages by slug, and the table or view a widget shows by UUID.
Size: 740 lines: 424 code, 297 tests and 19 generated OpenAPI.
What changes
targetType(tableorview) andtargetUuidto each widget'ssource, next to thenodeRelIdit already had. The old id still means nothing on another instance; the UUID does.importScheme, new optionalpagesparameter) applies each page of the scheme:importablePages()): only start pages and grid pages, and at most one start page, through one shared checkpage_<id>, in which case it gets a name of its own. An automatic slug never matches, so such a page is always added; matching on the application slug and the page slug together follows once Technical name for tables and applications, CRUD routes by application and table #3094 landsGridWidgetTypes::readWidget(), the reader the API and the user migration import use, with the uuid source set aside for the lookup below. A widget may only show a table or view that is a node of the application once the scheme's nodes are imported, the same rulePageServicefollows. The import still adds the nodes the scheme lists, as before, but a widget never adds a node by itself. The user migration import (importContext) didn't change.MenuServicetakes the page ids by the id the scheme gives them for this.compareSchemeChanges) returnspageswithfrom,to(the scheme pages with the kept widgets) andskipped. It counts as resources the nodes of the scheme the import would keep: ones the application has, ones the import adds, and ones that are here and manageable by the user. With the Resources switch off, the import checks the widgets again against the nodes that are sent. The menu preview no longer skips items for pages the import adds.Porting note: the frontend follows the Menu section of #3138. The fork branch
stack/s3-export-importwas used as a reference for the import rules (match by technical name, widgets replace the page's widgets), but its model (emoji, title-based names, a separate scheme service, grid views) doesn't apply upstream.This and #3202 both rework the node loop in
importScheme(), so whichever is merged second needs a small manual merge. It's simplest to merge #3202 first: this one then only adds$nodeIdsto the loop.How it was tested
After the reviews, on Nextcloud 34 with PostgreSQL:
ContextPagesRoundTripTest: an application with a table, a view, a grid page with three widgets and a page with an automatic name, plus a menu item on that page, is exported, previewed against an empty application and imported into it once it holds the scheme's two nodes. The pages and widgets arrive on the table and view the scheme's nodes resolve to, the automatically named page gets a new name and the menu item follows it, and two widgets are kept without a table or view and counted as such in the preview: one on a table the user can't manage, and one on a table that is here and manageable but isn't a node of the scheme. A second test previews a scheme whose table and view the import would add: widgets on them keep them, and a widget on a view that isn't here is kept without one.ContextServiceTest: the skip rules above, with their log linesphp-cs-fixer, clean; OpenAPI in step with the codeBy hand on a test instance (Nextcloud 34, PostgreSQL), through the API, 10 checks, all as expected: exported an application with a grid page with an automatic name, a data widget on its table, and a menu item on that page. I added a widget on a table that isn't on the instance to the file, then previewed and imported it into another application holding the same table. On that earlier version the preview listed that widget as skipped (it is now kept without a table); the import created the page under a new automatic name with its widget on the table's node there, the menu item followed the page, and no other table became a resource. The import screen was not tested by hand; the Cypress component test covers it.
🖼️ Screenshots
Screenshots follow in a comment.
🏁 Checklist
/backport to stableX.X🤖 AI (if applicable)