Repository navigation
feat(contexts): keep widgets in step with pages and applications - #3132
rubenvdlinde wants to merge 10 commits into
Conversation
9c0151e to
bcce8b6
Compare
bcce8b6 to
390968d
Compare
3dedb5e to
669999e
Compare
669999e to
af7e75b
Compare
enjeck
left a comment
There was a problem hiding this comment.
It seems like we have more test code than feature code. Most of the extra comes from the same case being tested in two or three places, or from tests that check mocks or config instead of behaviour. We should cut out some tests or merge. Having too many tests which are not useful slows down CI and adds clutter
| 'c.*', | ||
| 'r.id as node_rel_id', 'r.node_id', 'r.node_type', 'r.permissions', | ||
| 'p.id as page_id', 'p.page_type', | ||
| 'p.id as page_id', 'p.page_type', 'p.technical_name as page_technical_name', 'p.title as page_title', 'p.icon as page_icon', 'p.description as page_description', |
There was a problem hiding this comment.
Could we join only the start page here, and load the page fields in a separate query? Otherwise, imagine someone opens just a start page, we might be wasting time processing stuff that doesn't matter for them? Maybe i misunderstand
There was a problem hiding this comment.
You understood it right: every grid page multiplied the joined rows by the number of tables and shares. The base query now joins only the start page, as before this PR. findAll() and findById() load the other pages in one extra query for all applications in the result. findForNavBar() and findAllContainingNode() skip it, because the navigation shows names and the permission checks only read the nodes.
| $context->getDescription(), | ||
| array_values($nodes), | ||
| $context->getPages(), | ||
| $this->getPagesWithWidgets($context), |
There was a problem hiding this comment.
The scheme export now includes pages and widgets, but the scheme import ignores them though. Why?
There was a problem hiding this comment.
That was a gap: I added the export side and left the import for later. #3201 imports the pages and widgets of a scheme, with a Pages switch on the import screen next to the Menu one.
| // the relations of the table being imported, the only node a widget source can keep | ||
| $sourceRelMap = []; |
There was a problem hiding this comment.
We seem to have a bug: If I have an application with 2 tables and export an account. When i import, the application is created twice 🤔
There was a problem hiding this comment.
Thanks for finding this. It's older than this PR: the migrator called importContext() once per imported table and pointed every node at that one table. #3200 imports the tables and views first and then each application once, with every node on its own table or view.
| ['value' => 'left', 'label' => $this->l10n->t('Start')], | ||
| ['value' => 'center', 'label' => $this->l10n->t('Centered')], | ||
| ['value' => 'right', 'label' => $this->l10n->t('End')], |
There was a problem hiding this comment.
Maybe better if values are start, center and end too for consistency
There was a problem hiding this comment.
Done, the values are start, center and end now, with start as the default.
| * @throws Exception | ||
| * @throws \JsonException | ||
| */ | ||
| public function syncPage(int $pageId, array $widgets): void { |
There was a problem hiding this comment.
The PR says saving a page happens in one transaction, but syncPage() doesn't seem to. Do we need atomic()?
There was a problem hiding this comment.
Yes. It relied on its caller to open the transaction. syncPage() now runs in atomic() itself, and a test checks that a save failing halfway leaves the page as it was.
af7e75b to
dfede57
Compare
dfede57 to
d0fe0d6
Compare
|
About the tests: Same here: 40 test methods down to 25, and 1090 test lines down to 923. What's left is mostly database tests for deleting, unlinking and importing widgets, where a mock would only repeat the code. #3133 |
d0fe0d6 to
301f7fa
Compare
|
Hello there, We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process. Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6 Thank you for contributing to Nextcloud and we hope to hear from you soon! (If you believe you should not receive this message, you can add yourself to the blocklist.) |
3eca377 to
3184f77
Compare
3184f77 to
0fe8af1
Compare
|
@AndyScherzinger, one change since your approval: "fix(contexts): lock the application first when it is updated or deleted". Making the application update one transaction broke it on SQLite: a SQLite transaction that reads before it writes fails at once with "database is locked" when another request writes at the same time. That's what made the "Manage a context" Playwright tests fail on every retry on this stack.
|
|
@rubenvdlinde the approval here is from @blizzz (mine was a comment review), so blizzz should look at 0fe8af1 as well. The lock itself looks right to me. Writing first gets SQLite to wait for the lock instead of failing at once with "database is locked", and on MySQL/PostgreSQL One gap is left in Validated: reproduced on a deployed instance (MariaDB, #3133 at |
0fe8af1 to
8678cf3
Compare
|
Thanks @AndyScherzinger, and right, sorry: the approval is from @blizzz. @blizzz, could you look at "fix(contexts): lock the application first when it is updated or deleted" as well (4ca7656 since today's rebase)? It's explained in the comment above. The delete gap is fixed: when |
8678cf3 to
36ff89e
Compare
36ff89e to
14589f3
Compare
14589f3 to
aaa36fc
Compare
Saving a page syncs its widgets by id in one transaction. Deleting a page or an application deletes its widgets and their sources, and removing a table or view from an application deletes only the sources that point at it, so the widget stays. Moving a user's data carries pages, widgets and sources, remapped to the imported nodes and checked like a save. The header, text and data widget types are defined in one place with the schema their settings are checked against. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
- import and API read a widget with one GridWidgetTypes::readWidget() into a WidgetInput, so both apply the same rules; WidgetService takes those - an import skips a broken widget with a one-line warning instead of a trace, and inserts the widgets of a page with one source check - titles and descriptions of pages and widgets share TextHelper, which also treats no-break, ideographic and zero width spaces as blank; table, view and column titles use the same trim, and the length limits live there once - update() of an application checks the nodes it adds before it removes any, and runs in one transaction, so a refused request changes nothing - removing a node from an application deletes widget sources, page contents and the relation in the order deleteNodeRel() uses, with nothing swallowed - a save locks the widgets of its page and the relations it points at, and removing a relation locks it before its sources go, so neither leaves a stale source - a save writes a widget's source only when it changed - widget settings keep non-ASCII text and slashes unescaped, so the size limit counts the text as written - the boolean setting type is gone, as no widget type uses it - permission checks and the table and view lists load an application with its start page only; the other pages are loaded where they are read, in chunks - an account export loads the widgets of all applications with one query - WidgetServiceTest and ContextPagesRoundTripTest remove what an aborted run left Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
SQLite has no FOR UPDATE and refuses the query, while it runs one write transaction at a time anyway. Also the first class callable Rector asks for. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
On SQLite, a transaction that reads before it writes fails at once with "database is locked" when another request writes meanwhile; SQLite's wait for the lock does not apply then. Updating an application became one transaction that reads first, so parallel requests failed with a 500. ContextMapper::lock() locks the application row with SELECT ... FOR UPDATE, and on SQLite takes the write lock with a write that changes nothing, which waits for its turn. Update and delete call it first. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…t found Two deletes of one application at the same time both answered 200 and wrote two audit entries: the second waited for the lock and then deleted an application that was already gone. It now gets a 404. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…d audit it after the commit Inside the update's transaction, a failed write of the start page order was logged and skipped: the answer showed the new order while PostgreSQL rolled the whole update back. The update now fails instead, the audit entry is written after the commit, and it holds the permissions of added resources. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
… widget WidgetService stores the node relation of a data widget in the widget's own column. The check that a relation belongs to the application of the page and the lock taken before relations are removed move to ContextNodeRelationMapper. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…color Following enjeck's answers on #3099: a 'tables' widget type lists the tables and views of the application, and a header picks one color pair of the theme (primary, neutral, success, warning or error) instead of free background and text colors, so it stays readable in light, dark and high contrast themes. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
… hint The text widget is listed as Text, and a setting of a widget type can carry a hint that the form shows under its field. The text setting uses it to say that an empty line separates paragraphs. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
Since the update of an application accepts a node without permissions and gives an added one read, the audit entry of that node reads the same default instead of an undefined key. Assisted-by: ClaudeCode:claude-opus-5-5 Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
aaa36fc to
e171eec
Compare
Part of #3099, on top of #3131 (data model). This adds the behaviour #3100 describes for widgets, still without an API or UI.
Size: 1601 lines: 679 code and 922 tests.
What changes
syncPage()runs inatomic()), matched by id. Existing widgets are updated, new ones added, and missing ones removed.TablesMigrator) carries an application's pages, widgets and settings. A widget source only comes along when it points at the table being imported:importContext()maps every node to that table, so a source for another table would land on the wrong one. Imported widgets and page texts are checked the same way as saved ones, and invalid ones are skipped with a warning in the log. An application with more than one table still comes along once per table, an older bug that #C1 fixes.start,centerorend, and its colour is one colour pair of the theme:primary,neutral,success,warningorerror, so it stays readable in light, dark and high contrast. A setting can carry an optionalhint; the text setting uses it for the help text of its paragraph. A fourth type,tables, shows all tables and views of the application.Data exported with widgets can't be imported into an older version of Tables. That belongs in the changelog.
Three smaller changes come with it:
findAll()andfindById()load the other pages in one extra query for all applications in the result, so grid pages don't multiply the joined rows. The navigation bar and the permission checks don't read pages and skip that query.The widget types aren't in
ResponseDefinitionsyet, because the OpenAPI generator refuses schemas that no endpoint uses. They move there together with the endpoints in the next pull request.Why nothing breaks
There is still no way to create a widget, so all of this runs on empty tables. Deleting an application or removing a table behaves as before; it now also checks two more tables.
How it was tested
On Nextcloud 34 with PostgreSQL:
WidgetServiceTest(database, including clearing a widget's title, icon, description and settings, refusing a widget id from another page, and a save failing halfway leaving the page as it was)GridWidgetTypesTestandContextServiceTestContextPagesRoundTripTest(database: export, import, removing a table, deleting an application)ContextMapperTest(every page read byfindById()andfindAll()) andTablesMigratorTestWidgetServiceTestcompares the widgets as decoded arrays rather than JSON text, so it also holds on MySQL, whoseJSONcolumn reorders keyscomposer cs:check, psalm and rector, all clean🖼️ Screenshots
Backend only, no UI changes.
🏁 Checklist
/backport to stableX.X🤖 AI (if applicable)