Skip to content

feat(contexts): keep widgets in step with pages and applications - #3132

Open
rubenvdlinde wants to merge 10 commits into
mainfrom
feat/noid/grid-widget-lifecycle
Open

rubenvdlinde wants to merge 10 commits into
mainfrom
feat/noid/grid-widget-lifecycle

Conversation

@rubenvdlinde

@rubenvdlinde rubenvdlinde commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Part of #3099, on top of #3131 (data model). This adds the behaviour #3100 describes for widgets, still without an API or UI.

Size: 1601 lines: 679 code and 922 tests.

What changes

  • Saving a page replaces its widgets in one transaction (syncPage() runs in atomic()), matched by id. Existing widgets are updated, new ones added, and missing ones removed.
  • Deleting a page or an application deletes its widgets and their sources.
  • Removing a table or view from an application deletes only the source rows that point at it, first and in the same transaction as the rest. The widget stays, and later shows that it needs a new table.
  • Moving a user's data (TablesMigrator) carries an application's pages, widgets and settings. A widget source only comes along when it points at the table being imported: importContext() maps every node to that table, so a source for another table would land on the wrong one. Imported widgets and page texts are checked the same way as saved ones, and invalid ones are skipped with a warning in the log. An application with more than one table still comes along once per table, an older bug that #C1 fixes.
  • Exporting an application's scheme includes its pages and widgets; importing them follows in #C3.
  • Widget types (header, text, table or view) are defined in one place on the server, each with the schema its settings are checked against. A header's text alignment is start, center or end, and its colour is one colour pair of the theme: primary, neutral, success, warning or error, so it stays readable in light, dark and high contrast. A setting can carry an optional hint; the text setting uses it for the help text of its paragraph. A fourth type, tables, shows all tables and views of the application.
  • Limits: at most 100 widgets per page and 60,000 bytes of settings per widget. An import skips what goes over.
  • A widget's source must be a table or view of the page's own application. The service checks this itself, not only the API.

Data exported with widgets can't be imported into an older version of Tables. That belongs in the changelog.

Three smaller changes come with it:

  • Context responses now carry each page's title, icon, description and slug. The context query still joins only the start page; findAll() and findById() load the other pages in one extra query for all applications in the result, so grid pages don't multiply the joined rows. The navigation bar and the permission checks don't read pages and skip that query.
  • A grid page no longer gets an empty entry in the start page list.
  • Importing skips a start page entry whose table didn't come along. Before, it wrote an empty value into a column that doesn't allow one.

The widget types aren't in ResponseDefinitions yet, because the OpenAPI generator refuses schemas that no endpoint uses. They move there together with the endpoints in the next pull request.

Why nothing breaks

There is still no way to create a widget, so all of this runs on empty tables. Deleting an application or removing a table behaves as before; it now also checks two more tables.

How it was tested

On Nextcloud 34 with PostgreSQL:

  • WidgetServiceTest (database, including clearing a widget's title, icon, description and settings, refusing a widget id from another page, and a save failing halfway leaving the page as it was)
  • GridWidgetTypesTest and ContextServiceTest
  • ContextPagesRoundTripTest (database: export, import, removing a table, deleting an application)
  • ContextMapperTest (every page read by findById() and findAll()) and TablesMigratorTest
  • a broken export file and a two-table user migration
  • the full unit suite on the top of the stack (feat(contexts): carry the menu through export and import #3138), on PostgreSQL and SQLite: 684 tests. WidgetServiceTest compares the widgets as decoded arrays rather than JSON text, so it also holds on MySQL, whose JSON column reorders keys
  • composer cs:check, psalm and rector, all clean

🖼️ Screenshots

Backend only, no UI changes.

🏁 Checklist

  • ⛑️ Tests (unit and/or integration) are included or not needed
  • 🔙 Backport requests are created or not needed: /backport to stableX.X
  • 📅 Milestone is set
  • 🌸 PR title is meaningful (if it should be in the changelog: is it meaningful to users?)

🤖 AI (if applicable)

  • The content of this PR was partly or fully generated using AI

@rubenvdlinde
rubenvdlinde added this pull request to stack #3134 October 22, 2026 11:01
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from 9c0151e to bcce8b6 Compare October 22, 2026 11:37
@rubenvdlinde
rubenvdlinde removed this pull request from stack #3134 October 22, 2026 11:42
@rubenvdlinde
rubenvdlinde added this pull request to stack #3139 October 22, 2026 11:42
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from bcce8b6 to 390968d Compare October 22, 2026 13:25
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch 2 times, most recently from 3dedb5e to 669999e Compare October 24, 2026 08:09
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from 669999e to af7e75b Compare October 26, 2026 09:24
@rubenvdlinde
rubenvdlinde marked this pull request as ready for review October 26, 2026 09:25

@enjeck enjeck left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like we have more test code than feature code. Most of the extra comes from the same case being tested in two or three places, or from tests that check mocks or config instead of behaviour. We should cut out some tests or merge. Having too many tests which are not useful slows down CI and adds clutter

Comment thread lib/Db/ContextMapper.php Outdated
'c.*',
'r.id as node_rel_id', 'r.node_id', 'r.node_type', 'r.permissions',
'p.id as page_id', 'p.page_type',
'p.id as page_id', 'p.page_type', 'p.technical_name as page_technical_name', 'p.title as page_title', 'p.icon as page_icon', 'p.description as page_description',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we join only the start page here, and load the page fields in a separate query? Otherwise, imagine someone opens just a start page, we might be wasting time processing stuff that doesn't matter for them? Maybe i misunderstand

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You understood it right: every grid page multiplied the joined rows by the number of tables and shares. The base query now joins only the start page, as before this PR. findAll() and findById() load the other pages in one extra query for all applications in the result. findForNavBar() and findAllContainingNode() skip it, because the navigation shows names and the permission checks only read the nodes.

$context->getDescription(),
array_values($nodes),
$context->getPages(),
$this->getPagesWithWidgets($context),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The scheme export now includes pages and widgets, but the scheme import ignores them though. Why?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was a gap: I added the export side and left the import for later. #3201 imports the pages and widgets of a scheme, with a Pages switch on the import screen next to the Menu one.

Comment on lines +629 to +630
// the relations of the table being imported, the only node a widget source can keep
$sourceRelMap = [];

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We seem to have a bug: If I have an application with 2 tables and export an account. When i import, the application is created twice 🤔

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for finding this. It's older than this PR: the migrator called importContext() once per imported table and pointed every node at that one table. #3200 imports the tables and views first and then each application once, with every node on its own table or view.

Comment thread lib/Service/GridWidgetTypes.php Outdated
Comment on lines +74 to +76
['value' => 'left', 'label' => $this->l10n->t('Start')],
['value' => 'center', 'label' => $this->l10n->t('Centered')],
['value' => 'right', 'label' => $this->l10n->t('End')],

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe better if values are start, center and end too for consistency

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, the values are start, center and end now, with start as the default.

* @throws Exception
* @throws \JsonException
*/
public function syncPage(int $pageId, array $widgets): void {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR says saving a page happens in one transaction, but syncPage() doesn't seem to. Do we need atomic()?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. It relied on its caller to open the transaction. syncPage() now runs in atomic() itself, and a test checks that a save failing halfway leaves the page as it was.

@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from af7e75b to dfede57 Compare November 2, 2026 11:17
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from dfede57 to d0fe0d6 Compare November 2, 2026 11:23
@rubenvdlinde

Copy link
Copy Markdown
Contributor Author

About the tests: Same here: 40 test methods down to 25, and 1090 test lines down to 923. What's left is mostly database tests for deleting, unlinking and importing widgets, where a mock would only repeat the code.

#3133

@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from d0fe0d6 to 301f7fa Compare November 3, 2026 14:51
@rubenvdlinde
rubenvdlinde removed this pull request from stack #3139 November 4, 2026 10:41
@rubenvdlinde
rubenvdlinde added this pull request to stack #3222 November 5, 2026 09:14
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Hello there,
Thank you so much for taking the time and effort to create a pull request to our Nextcloud project.

We hope that the review process is going smooth and is helpful for you. We want to ensure your pull request is reviewed to your satisfaction. If you have a moment, our community management team would very much appreciate your feedback on your experience with this PR review process.

Your feedback is valuable to us as we continuously strive to improve our community developer experience. Please take a moment to complete our short survey by clicking on the following link: https://cloud.nextcloud.com/apps/forms/s/i9Ago4EQRZ7TWxjfmeEpPkf6

Thank you for contributing to Nextcloud and we hope to hear from you soon!

(If you believe you should not receive this message, you can add yourself to the blocklist.)

@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from 3eca377 to 3184f77 Compare November 13, 2026 21:03
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from 3184f77 to 0fe8af1 Compare November 16, 2026 09:22
@rubenvdlinde

Copy link
Copy Markdown
Contributor Author

@AndyScherzinger, one change since your approval: "fix(contexts): lock the application first when it is updated or deleted".

Making the application update one transaction broke it on SQLite: a SQLite transaction that reads before it writes fails at once with "database is locked" when another request writes at the same time. That's what made the "Manage a context" Playwright tests fail on every retry on this stack.

ContextMapper::lock() now locks the application row as the first statement of an update and a delete. On MySQL and PostgreSQL that's SELECT ... FOR UPDATE; on SQLite it's a write that changes nothing, which waits for its turn instead of failing. With it, the Playwright tests ran 35 times on SQLite without a failure. #3133 uses the same lock when a page is added, instead of its own PageMapper::lockContext().

@AndyScherzinger

Copy link
Copy Markdown
Member

@rubenvdlinde the approval here is from @blizzz (mine was a comment review), so blizzz should look at 0fe8af1 as well.

The lock itself looks right to me. Writing first gets SQLite to wait for the lock instead of failing at once with "database is locked", and on MySQL/PostgreSQL FOR UPDATE makes concurrent saves of one application run one after the other. That also covers the concurrent half of #3255 and the lock suggested in #3264.

One gap is left in delete(). It reads the application before the transaction and ignores lock() returning false. So when two deletes of the same application arrive together, the second one waits, then runs its deletes on an application that is already gone, answers 200 and writes a second "Tables application with ID … was deleted" audit entry. Throwing NotFoundError when lock() returns false (as PageService::create() does) closes it.

Validated: reproduced on a deployed instance (MariaDB, #3133 at 1d7e6ea5, so with 0fe8af1; two parallel DELETE /api/2/contexts/{id} as alice, 12 runs: in 5 runs both answered 200 and audit.log has two deletion entries for that application; in the other 7 the second answered 404).

@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from 0fe8af1 to 8678cf3 Compare November 17, 2026 09:11
@rubenvdlinde

Copy link
Copy Markdown
Contributor Author

Thanks @AndyScherzinger, and right, sorry: the approval is from @blizzz. @blizzz, could you look at "fix(contexts): lock the application first when it is updated or deleted" as well (4ca7656 since today's rebase)? It's explained in the comment above.

The delete gap is fixed: when lock() finds the application gone, delete() now throws NotFoundError, so the second of two parallel deletes answers 404 and writes no audit entry. There's a test for it.

@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from 8678cf3 to 36ff89e Compare November 19, 2026 08:27
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from 36ff89e to 14589f3 Compare November 19, 2026 13:35
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from 14589f3 to aaa36fc Compare November 20, 2026 10:52
Base automatically changed from feat/noid/grid-page-model to main November 20, 2026 13:19
Saving a page syncs its widgets by id in one transaction. Deleting a
page or an application deletes its widgets and their sources, and
removing a table or view from an application deletes only the sources
that point at it, so the widget stays.

Moving a user's data carries pages, widgets and sources, remapped to
the imported nodes and checked like a save. The header, text and data
widget types are defined in one place with the schema their settings
are checked against.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
- import and API read a widget with one GridWidgetTypes::readWidget() into a
  WidgetInput, so both apply the same rules; WidgetService takes those
- an import skips a broken widget with a one-line warning instead of a trace, and
  inserts the widgets of a page with one source check
- titles and descriptions of pages and widgets share TextHelper, which also treats
  no-break, ideographic and zero width spaces as blank; table, view and column titles
  use the same trim, and the length limits live there once
- update() of an application checks the nodes it adds before it removes any, and
  runs in one transaction, so a refused request changes nothing
- removing a node from an application deletes widget sources, page contents and the
  relation in the order deleteNodeRel() uses, with nothing swallowed
- a save locks the widgets of its page and the relations it points at, and removing
  a relation locks it before its sources go, so neither leaves a stale source
- a save writes a widget's source only when it changed
- widget settings keep non-ASCII text and slashes unescaped, so the size limit counts
  the text as written
- the boolean setting type is gone, as no widget type uses it
- permission checks and the table and view lists load an application with its start
  page only; the other pages are loaded where they are read, in chunks
- an account export loads the widgets of all applications with one query
- WidgetServiceTest and ContextPagesRoundTripTest remove what an aborted run left

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
SQLite has no FOR UPDATE and refuses the query, while it runs one write
transaction at a time anyway. Also the first class callable Rector asks for.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
On SQLite, a transaction that reads before it writes fails at once with
"database is locked" when another request writes meanwhile; SQLite's wait
for the lock does not apply then. Updating an application became one
transaction that reads first, so parallel requests failed with a 500.

ContextMapper::lock() locks the application row with SELECT ... FOR UPDATE,
and on SQLite takes the write lock with a write that changes nothing, which
waits for its turn. Update and delete call it first.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…t found

Two deletes of one application at the same time both answered 200 and wrote
two audit entries: the second waited for the lock and then deleted an
application that was already gone. It now gets a 404.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…d audit it after the commit

Inside the update's transaction, a failed write of the start page order was
logged and skipped: the answer showed the new order while PostgreSQL rolled
the whole update back. The update now fails instead, the audit entry is
written after the commit, and it holds the permissions of added resources.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
… widget

WidgetService stores the node relation of a data widget in the widget's own
column. The check that a relation belongs to the application of the page and
the lock taken before relations are removed move to ContextNodeRelationMapper.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
…color

Following enjeck's answers on #3099: a 'tables' widget type lists the tables and
views of the application, and a header picks one color pair of the theme
(primary, neutral, success, warning or error) instead of free background and
text colors, so it stays readable in light, dark and high contrast themes.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
… hint

The text widget is listed as Text, and a setting of a widget type can carry
a hint that the form shows under its field. The text setting uses it to say
that an empty line separates paragraphs.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
Since the update of an application accepts a node without permissions and
gives an added one read, the audit entry of that node reads the same default
instead of an undefined key.

Assisted-by: ClaudeCode:claude-opus-5-5
Signed-off-by: Ruben van der Linde <ruben@conduction.nl>
@rubenvdlinde
rubenvdlinde force-pushed the feat/noid/grid-widget-lifecycle branch from aaa36fc to e171eec Compare November 21, 2026 10:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants