v0.9.7: embeddings cleanup, search expansion, navigation and execution speedup - #8439
Open
waleedlatif1 wants to merge 14 commits into
Open
waleedlatif1 wants to merge 14 commits into
waleedlatif1 wants to merge 14 commits into
Conversation
…8421) * improvement(desktop): surface update actions in a persistent toast * fix(desktop): distinguish update toast dismissal from eviction
…oute reload/back to the browser tab (#8429) * fix(desktop-browser): keep the chat caret while the agent works and route reload/back to the browser tab * fix(desktop-browser): return popup focus to the user's page and wait for a shown view
* feat(cli): add sim cli search and default output to JSON - sim cli search ranks every command in the tree against a plain-language query, locally via minisearch, and returns the top 5 - root and group --help open with a discovery note when a coding agent runs the CLI, pointing it at sim cli search - default output format is now json; table stays available via --output, SIM_OUTPUT, or sim configure --set-output - reword command descriptions that lacked the words people search with * fix(cli): describe billing status by what it reports, scope search privacy claim to the query * chore(cli): release 2.2.0
* fix(search): retire legacy embeddings through scoped backfill * fix(search): run retirement without cleanup flags and close ingestion gaps * chore(tests): scope migration recovery journal assertion * chore(tests): align document dispatch billing and quota fixtures
…ts first block (#8434) * improvement(execution): cut the time between a workflow trigger and its first block - Trigger.dev getJob only retrieves real run ids; caller-chosen ids (schedule_…, workflow-execution:…) go straight to the tag lookup instead of a ~10 s 404 - Execution-log start resolves the snapshot id (cached once referenced) and inserts with ON CONFLICT instead of select + full-state upsert RETURNING * + insert - Preprocessing starts payer attribution and the ban/usage/subscription gate reads as soon as their inputs are known; subscription is only read when needed - Execution core prefetches env + PII policy alongside custom blocks and reuses the webhook job's already-loaded environment - Webhook lookups answer trigger-block deployment from the join they already do; versioned deployment loads serve the materialized cache first - getHighestPrioritySubscription, custom-block rows, and env suspension lookups drop sequential round trips * fix(execution): scope the snapshot FK retry, keep a suspended identity's access read out of the run, tidy tests * improvement(execution): encapsulate early admission reads, plain snapshot identity, explicit suspension paths * improvement(environment): reuse the actor's resolved access when withholding a suspended identity
…switches (#8435) * improvement(settings): remove the 300ms floor under settings section switches Every section switch mounted a fresh Suspense boundary (the empty section loading.tsx and a page-level <Suspense fallback={null}> around the code-split body). React 19 holds content that resolves into a just-committed fallback for at least 300ms, so every switch paid that before the section even rendered or started its queries. - drop the section loading boundaries and page-level Suspense on all settings planes; navigations are transitions, so the outgoing section stays until the incoming one is ready - move the sidebar selection on click so the click still reads as acknowledged - warm each hot section's first-content queries on navigation intent - load the fork sync editor and custom tool editor on open; they pulled the block and trigger registries into the list chunks * fix(settings): tie the pending sidebar selection to its navigation The pending row cleared only on a pathname change, so a navigation the server redirected back to the current section, or one that failed, left the clicked row selected and its click guard swallowing retries. Set the selection optimistically inside the navigation's own transition so React drops it when that transition settles.
…ion (#8431) * feat(logs): allow omitting workflow snapshots from diagnostic reads * feat(stripe): add optional subscription pagination cursors * fix: clarify Stripe guidance and log snapshot defaults
* fix(search): finish retirement and index maintenance on deploy * docs(search): clarify retirement migration session requirement
* improvement(chat): show live search sources by default * test(chat): cover live search disclosure transitions --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+36
to
+40
| const targets = await tx<{ id: string }[]>` | ||
| SELECT id FROM knowledge_base WHERE is_search_index LIMIT 2` | ||
| if (targets.length === 0) return null | ||
| if (targets.length > 1) { | ||
| throw new Error('Multiple Search knowledge bases found; cleanup target is ambiguous') |
Contributor
There was a problem hiding this comment.
The database allows one Search index per workspace or organization, so a deployment can have more than one. In that case, this check throws instead of retiring them. Migration 0027 then remains incomplete, and the deployment cannot finish until the migration handles multiple indexes.
Knowledge Base Used: Database schema and migrations
Comment on lines
+35
to
+37
| if (existing.length === 0) { | ||
| const targets = await tx<{ id: string }[]>` | ||
| SELECT id FROM knowledge_base WHERE is_search_index LIMIT 2` |
Contributor
There was a problem hiding this comment.
If a deployment sets SIM_SEARCH_LIVE=false, indexed Search still uses these knowledge bases. This migration runs regardless of that setting, disables their documents, and deletes their embeddings before the new application starts. Search then loses its indexed results; changing the setting back cannot restore them without reindexing.
Knowledge Base Used:
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
4 issues found across 149 files
Confidence score: 2/5
packages/db/script-migrations/0027_retire_search_embeddings.tscan delete embeddings still needed whileSIM_SEARCH_LIVE=false, leaving indexed Search results unavailable even if the feature is re-enabled; defer retirement until the flag is enabled.apps/sim/background/webhook-execution.tscan reuse a pre-resolved environment after access is revoked or the environment is suspended, exposing old secrets to workflow blocks; recheck access and suspension before execution.apps/sim/ee/workspace-forking/components/fork-sync-detail-view/fork-sync-detail-view.tsxlets users switch Push/Pull and lose unsaved mapping edits without confirmation; apply the unsaved-changes guard to the direction switch.apps/sim/lib/knowledge/documents/service.tscan leave a pending row and charged attempt when a worker exits after the knowledge base becomes dormant; withdraw or refund the affected generation when returning early.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/ee/workspace-forking/components/fork-sync-detail-view/fork-sync-detail-view.tsx">
<violation number="1" location="apps/sim/ee/workspace-forking/components/fork-sync-detail-view/fork-sync-detail-view.tsx:117">
P2: Switching Push/Pull bypasses the unsaved-changes guard, and the direction change clears the current mapping edits. Guard this switch so users can confirm before losing their changes.</violation>
</file>
<file name="apps/sim/background/webhook-execution.ts">
<violation number="1" location="apps/sim/background/webhook-execution.ts:1171">
P2: This lets `executeWorkflowCore` reuse an environment resolved before webhook setup and input processing, skipping its access and suspension checks; revocation during that gap still exposes the old secrets to blocks. Revalidate before reuse or let the executor load its own snapshot.</violation>
</file>
<file name="apps/sim/lib/knowledge/documents/service.ts">
<violation number="1" location="apps/sim/lib/knowledge/documents/service.ts:1676">
P2: A queued worker can reach this return after the KB becomes dormant, leaving its pending row with the queue token, timestamp, and charged attempt even though the task exits normally. Withdraw or refund that exact generation before returning so later dispatches are not held behind a stale queue stamp.</violation>
</file>
<file name="packages/db/script-migrations/0027_retire_search_embeddings.ts">
<violation number="1" location="packages/db/script-migrations/0027_retire_search_embeddings.ts:37">
P1: Defer this retirement while `SIM_SEARCH_LIVE=false`; the unconditional cleanup disables documents and deletes embeddings that indexed Search still needs, and re-enabling it cannot restore those results without reindexing.</violation>
</file>
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
| >`SELECT * FROM search_embedding_cleanup_progress WHERE id = 1` | ||
| if (existing.length === 0) { | ||
| const targets = await tx<{ id: string }[]>` | ||
| SELECT id FROM knowledge_base WHERE is_search_index LIMIT 2` |
Contributor
There was a problem hiding this comment.
P1: Defer this retirement while SIM_SEARCH_LIVE=false; the unconditional cleanup disables documents and deletes embeddings that indexed Search still needs, and re-enabling it cannot restore those results without reindexing.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At packages/db/script-migrations/0027_retire_search_embeddings.ts, line 37:
<comment>Defer this retirement while `SIM_SEARCH_LIVE=false`; the unconditional cleanup disables documents and deletes embeddings that indexed Search still needs, and re-enabling it cannot restore those results without reindexing.</comment>
<file context>
@@ -0,0 +1,174 @@
+ >`SELECT * FROM search_embedding_cleanup_progress WHERE id = 1`
+ if (existing.length === 0) {
+ const targets = await tx<{ id: string }[]>`
+ SELECT id FROM knowledge_base WHERE is_search_index LIMIT 2`
+ if (targets.length === 0) return null
+ if (targets.length > 1) {
</file context>
| > | ||
| <ForkSyncView | ||
| controller={controller} | ||
| onDirectionChange={(next) => void setDirection(next)} |
Contributor
There was a problem hiding this comment.
P2: Switching Push/Pull bypasses the unsaved-changes guard, and the direction change clears the current mapping edits. Guard this switch so users can confirm before losing their changes.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/sim/ee/workspace-forking/components/fork-sync-detail-view/fork-sync-detail-view.tsx, line 117:
<comment>Switching Push/Pull bypasses the unsaved-changes guard, and the direction change clears the current mapping edits. Guard this switch so users can confirm before losing their changes.</comment>
<file context>
@@ -0,0 +1,202 @@
+ >
+ <ForkSyncView
+ controller={controller}
+ onDirectionChange={(next) => void setDirection(next)}
+ />
+ </SettingsPanel>
</file context>
Collaborator
Author
There was a problem hiding this comment.
Fixed in #8440: the direction switch now goes through the view's unsaved-changes guard, so unsaved mapping edits confirm before the switch drops them. Without edits it switches immediately, as before.
| loggingSession, | ||
| trustedInitialResolvedSecretTraceProvenance: | ||
| resolvedSecretTraceRegistry.exportProvenanceForValue(triggerInput), | ||
| preloadedEnvironment, |
Contributor
There was a problem hiding this comment.
P2: This lets executeWorkflowCore reuse an environment resolved before webhook setup and input processing, skipping its access and suspension checks; revocation during that gap still exposes the old secrets to blocks. Revalidate before reuse or let the executor load its own snapshot.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/sim/background/webhook-execution.ts, line 1171:
<comment>This lets `executeWorkflowCore` reuse an environment resolved before webhook setup and input processing, skipping its access and suspension checks; revocation during that gap still exposes the old secrets to blocks. Revalidate before reuse or let the executor load its own snapshot.</comment>
<file context>
@@ -1151,6 +1168,7 @@ async function executeWebhookJobInternal(
loggingSession,
trustedInitialResolvedSecretTraceProvenance:
resolvedSecretTraceRegistry.exportProvenanceForValue(triggerInput),
+ preloadedEnvironment,
includeFileBase64: false,
base64MaxBytes: undefined,
</file context>
| .limit(1) | ||
|
|
||
| if (contextRows[0] && !requiresConnectorIndexing(contextRows[0].isSearchIndex)) { | ||
| return { outcome: 'skipped', reason: 'unavailable' } |
Contributor
There was a problem hiding this comment.
P2: A queued worker can reach this return after the KB becomes dormant, leaving its pending row with the queue token, timestamp, and charged attempt even though the task exits normally. Withdraw or refund that exact generation before returning so later dispatches are not held behind a stale queue stamp.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At apps/sim/lib/knowledge/documents/service.ts, line 1676:
<comment>A queued worker can reach this return after the KB becomes dormant, leaving its pending row with the queue token, timestamp, and charged attempt even though the task exits normally. Withdraw or refund that exact generation before returning so later dispatches are not held behind a stale queue stamp.</comment>
<file context>
@@ -1653,6 +1672,9 @@ export async function processDocumentAsync(
.limit(1)
+ if (contextRows[0] && !requiresConnectorIndexing(contextRows[0].isSearchIndex)) {
+ return { outcome: 'skipped', reason: 'unavailable' }
+ }
if (contextRows.length === 0) {
</file context>
…er and route hard reload through recovery (#8442) * fix(desktop-browser): keep error-page focus recovery inside the browser and route hard reload through recovery * fix(desktop-browser): recover issue-page focus only when nothing holds focus, and keep cache bypass except for hung pages * fix(desktop-browser): retry the failed URL when hard reloading a load error
* fix(search): retire indexed content across all Search KBs * fix(search): validate retirement scope through completion
* feat(planetscale): add management actions * fix(planetscale): support default-parent backups and input limits * fix(planetscale): retain active backup branch context * fix(planetscale): address integration validation findings * chore(planetscale): clarify input and restore guidance --------- Co-authored-by: Bill Leoutsakos <billleoutsakos@Bills-MacBook-Pro.local>
…warm every section (#8440) * improvement(settings): paint settings section clicks immediately and cut a round trip - Share the in-flight section navigation between the sidebar and the content area: the clicked section's heading paints over an empty body while its route resolves, with the outgoing section kept laid out but invisible and inert. It is optimistic state inside the navigation's own transition, not a Suspense fallback, so nothing is held back and a redirect back restores the section untouched. - A press warms data and code but no longer prefetches the route: a prefetch started at mousedown made the navigation wait on a second, two-step request instead of its own. - Warm every section's chunk on intent from the persistent settings layout instead of the sidebar, so the chunk loads alongside the route payload instead of after it, without growing the workspace chrome's module graph. - Seed fork availability for admins and read the server-seeded viewer permission, so Workspace Forks renders with the rest of the sidebar instead of after a session fetch and an availability fetch. * fix(settings): keep press prefetch, guard fork direction switch, tighten docs - Restore the sidebar link's press-time route prefetch. Measured with a realistic ~90ms press, the prefetch mousedown starts completes before the click commits, so removing it slowed every navigation (commit ~105ms -> ~190ms after mousedown). - Route the fork sync direction switch through the unsaved-changes guard: switching Push/Pull drops the in-session mapping choices, so unsaved edits confirm first. - Give SettingsNavigationProvider a props interface, and scope the section layout's 404/307 note to section authorization, which is what an unknown or legacy segment skips. * fix(settings): confirm every discarded fork choice, keep settings UI out of the workspace chrome - A fork sync direction switch drops every in-session choice, not just unsaved mapping edits. Expose hasSessionChoices (mapping edits, a copy selection that differs from the default, accepted dropped references, trigger URL choices) and confirm the switch whenever any is set. - Move SettingsPendingSection into its own module so the provider in the workspace chrome no longer imports the settings header UI. * fix(fork-sync): clear committed session choices after a sync A successful sync applies the accepted drops and trigger URL choices, so clear exactly the snapshots it submitted, as it already does for mapping edits; choices made while the request was in flight stay. Compare the copy selection over the visible candidates only, the ones a sync sends, so keys left behind by a completed copy never count as a change. * chore(settings): rebase module-graph baselines onto staging Record the settings layout's section import() edges and the workspace chrome's smaller graph against the current staging baseline.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.