Skip to content

Commit ba485c6

Browse files
authored
improvement(logs): cut recurring production log noise and fix the bugs behind it (#8871)
* fix(serializer): skip tool selection for trigger-mode blocks Trigger-mode blocks run through TriggerBlockHandler and never read a tool id, while their tool-mode sub-blocks (e.g. operation) are not serialized, so selecting a tool always threw and logged a fallback warning. * improvement(executor): memoize block output schemas per serialized block collectBlockData re-derived every block's output schema on each condition, function, agent, and reference evaluation, re-parsing an invalid agent response format and logging its full stack every time. Memoize the schema per immutable serialized block, attribute parse failures to the real block id, and log them at debug without the stack. * fix(webhooks): stop warning on accountless webhook credentials Service-account and managed-OAuth credentials (e.g. a custom Slack bot) have no OAuth account row, so the account-owner lookup always missed and warned on every webhook run. Skip the lookup for them and warn only when the credential or its account is genuinely missing. * improvement(webhooks): log Slack reactions.get missing_scope once per webhook A Slack bot without reactions:read fails reactions.get identically for every reaction event until it is reinstalled. Record that configuration state once per webhook per hour at info and keep the warning for other failures. * improvement(workflows): repair drifted subBlock types quietly Deployed snapshots are immutable and re-sanitized on every load, so a field whose declared type later changed (e.g. dropdown to combobox) warned on every materialization forever. Log repairs at debug when the stored type is one some registered block declares; keep warning for missing or unknown types. * improvement(catalog): skip custom block input derivation for tool reads Listing, reading, and executing a catalog tool resolved the catalog gate with every custom block's input fields, which loads and sanitizes each custom block's deployed workflow per request. Tool scope only needs the block types (every custom block exposes workflow_executor), so tool reads use the lightweight overlay rows; block reads still derive the inputs. * improvement(webhooks): debug-log missing idempotency ids for providers without one A generic webhook without an idempotency header or configured field has no delivery identifier by design, so warning on every delivery was noise. Warn only when the provider normally extracts an id from the body. * fix(flint): declare generate_pages items as an array param items was typed json, so the LLM schema advertised it as an object and dropped its item schema with a warning on every schema build. Declare it as an array of 1-10 page objects; block-provided JSON strings still parse through parseItems. Regenerate tool metadata and docs. * improvement(executor): skip JSON parsing for json inputs holding plain strings json-typed block inputs also accept plain strings (a file id, URL, or reference) that are kept as-is, so parsing them could only fail and warn. Parse only text that can be JSON; every valid JSON text still parses. * improvement(mothership): classify catalog routes and dedupe missing-provenance warnings Catalog responses (blocks, tools, connector types) carry no workspace data, so no producer records provenance for them and the per-request warning was noise. Skip them statically, and report a data-bearing route without a producer once per route per hour instead of on every request. Admission and recording behavior is unchanged. * fix(tokenization): resolve tiktoken encodings by model family encodingForModel only knows exact OpenAI names, so newer or provider-prefixed OpenAI ids fell back to cl100k_base (miscounting o200k models) and every unknown model built and cached its own copy of the rank table. Resolve exact names through js-tiktoken, newer OpenAI ids by family, everything else to cl100k_base, and cache one instance per encoding. * improvement(proxy): log blocked scanner requests at debug Every blocked request is an empty or tool user agent probing paths like /admin.php; the 403 is the whole response, so the per-request warning was pure ingest cost. * improvement(telemetry): downgrade best-effort collector failures to warn Forwarding to the external collector is best effort and the route still returns success, so a collector timeout or error is not an application error. * fix(admin): log mothership admin proxy failures and map upstream 5xx to 502 The proxy logged nothing, passed an upstream 5xx straight through as its own 500, threw on a non-JSON upstream body, and returned a silent 500 when the admin key was missing. Log the method, environment, endpoint, status, and a truncated upstream body; answer upstream 5xx and unparseable bodies with 502; and log the missing-key misconfiguration. * improvement(logs): move per-call success chatter to debug PII batch masking, function execution request/success, sandbox mount resolution, table row queries and limits, usage-limit statistics, DAG builds, and mothership tool routing logged an info line on every call. They are diagnostic detail, not events, so log them at debug. * chore(audits): shrink explicit-any baseline after tokenization cleanup * improvement(mothership): keep tool call routing decision at info It is the diagnostic for stale-catalog 'No handler for tool' dispatches and in-band double-dispatch races, so it stays visible in production. * fix(admin): keep upstream message and empty 2xx bodies in the mothership proxy A gateway 502 now carries the upstream's error or message field, which the admin UI shows, and an empty 2xx body (e.g. 204) passes through as an empty response instead of failing JSON serialization into a misleading 502. * improvement(tokenization): memoize model to encoding name resolution A non-OpenAI model threw and caught inside getEncodingNameForModel on every count. Cache the resolved encoding name per model id in a bounded LRU. * improvement(mothership): resolve catalog route patterns through the route table Derive each catalog pattern by matching its contract path against the generated v2 route table, so a renamed path parameter cannot drift from the pattern matchV2Route reports and silently restore the warning. * fix(webhooks): warn on missing idempotency ids unless the provider declares them optional Header-based providers (GitHub, GitLab, Shopify, Linear, Svix) have no body extractor, so keying the warning on the extractor silenced exactly the anomalous case of their delivery header going missing. Providers now declare deliveryIdOptional; only the generic webhook does, since its deliveries carry no identifier unless an idempotency field is configured. * refactor(logs): simplify log-noise fixes after review - Skip reactions.get entirely while a webhook's bot is known to lack the scope, instead of repeating a call known to fail. - Treat any intact sub-block whose stored type differs from the registry as drift, dropping the registry-wide type scan. - Move the JSON-candidate check next to isJSONString and trim once. - Make the catalog gate's cheap overlay rows the default; only block reads derive custom block inputs. - Collapse the admin mothership proxy's three copied handlers into one. * chore(logs): tighten comments and rename trigger-mode serializer test * test(mothership): guard sandbox catalog route classification against the route table Extract the catalog route set into isCatalogRoute, built lazily from the catalog contracts resolved through the generated v2 route table, and test it against real request paths so a route or parameter rename cannot silently restore the missing-provenance warning. * fix(webhooks): keep calling reactions.get after a missing-scope report Skipping the call for the cooldown window left a reinstalled Slack bot with empty reaction text for up to an hour. Keep the once-per-window log but always make the call. Block listing also uses the lightweight custom-block rows, since block summaries never read custom block inputs. * chore(logs): tighten log-noise changes after a line-by-line audit - Drop the blockId option threaded through getEffectiveBlockOutputs; it only labeled a debug log. - Name the log-dedupe windows and cache ceilings. - Build the sandbox catalog route set by converting contract paths to the route table's pattern syntax; the route-table test guards drift.
1 parent 7dc251a commit ba485c6

36 files changed

Lines changed: 435 additions & 186 deletions

File tree

‎apps/docs/content/docs/integrations/flint.mdx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -56,7 +56,7 @@ Start a background Flint agent task that generates up to 10 pages from a templat
5656
| `apiKey` | string | Yes | Flint API key \(found in Flint team settings, starts with ak_\) |
5757
| `siteId` | string | Yes | ID of the Flint site the agent should modify |
5858
| `templatePageSlug` | string | Yes | Slug of the existing template page to generate from \(e.g., /case-studies/template\) |
59-
| `items` | json | Yes | JSON array of 1-10 pages to generate. Each item requires targetPageSlug \(slug for the new page\) and context \(content details the agent should use\). |
59+
| `items` | array | Yes | JSON array of 1-10 pages to generate. Each item requires targetPageSlug \(slug for the new page\) and context \(content details the agent should use\). |
6060
| `callbackUrl` | string | No | HTTPS webhook URL that Flint will POST to when the task completes or fails |
6161
| `publish` | boolean | No | Whether to automatically publish the generated pages when the task completes |
6262

Lines changed: 95 additions & 119 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
import { db } from '@sim/db'
22
import { settings, user } from '@sim/db/schema'
3+
import { createLogger } from '@sim/logger'
34
import { getErrorMessage } from '@sim/utils/errors'
5+
import { isRecordLike } from '@sim/utils/object'
6+
import { truncate } from '@sim/utils/string'
47
import { eq } from 'drizzle-orm'
58
import { type NextRequest, NextResponse } from 'next/server'
69
import { adminMothershipQuerySchema } from '@/lib/api/contracts/mothership-chats'
@@ -11,6 +14,10 @@ import { env } from '@/lib/core/config/env'
1114
import { withRouteHandler } from '@/lib/core/utils/with-route-handler'
1215
import { getMothershipBaseURL } from '@/lib/mothership/server/agent-url'
1316

17+
const logger = createLogger('AdminMothershipProxy')
18+
19+
const UPSTREAM_BODY_LOG_LIMIT = 500
20+
1421
const ENV_URLS: Record<string, string | undefined> = {
1522
dev: env.MOTHERSHIP_DEV_URL,
1623
staging: env.MOTHERSHIP_STAGING_URL,
@@ -54,77 +61,106 @@ async function getAuthorizedAdminUserId() {
5461
return authorized ? session.user.id : null
5562
}
5663

57-
/**
58-
* Proxy to the mothership admin API.
59-
*
60-
* Query params:
61-
* env - "dev" | "staging" | "prod"
62-
* endpoint - the admin endpoint path, e.g. "requests", "licenses", "traces"
63-
*
64-
* The request body (for POST) is forwarded as-is. Additional query params
65-
* (e.g. requestId for GET /traces) are forwarded.
66-
*/
67-
export const POST = withRouteHandler(async (req: NextRequest) => {
68-
const userId = await getAuthorizedAdminUserId()
69-
if (!userId) {
70-
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
71-
}
72-
73-
const adminKey = env.MOTHERSHIP_API_ADMIN_KEY
74-
if (!adminKey) {
75-
return NextResponse.json({ error: 'MOTHERSHIP_API_ADMIN_KEY not configured' }, { status: 500 })
76-
}
77-
78-
const { searchParams } = new URL(req.url)
79-
const queryValidation = adminMothershipQuerySchema.safeParse(searchParamsToObject(searchParams))
80-
if (!queryValidation.success) return validationErrorResponse(queryValidation.error)
81-
const { env: environment, endpoint } = queryValidation.data
82-
83-
if (!isValidEndpoint(endpoint)) {
84-
return NextResponse.json({ error: 'invalid endpoint' }, { status: 400 })
64+
/** `undefined` when the body is not JSON. Only called on a non-empty body. */
65+
function parseJsonText(text: string): unknown {
66+
try {
67+
return JSON.parse(text)
68+
} catch {
69+
return undefined
8570
}
71+
}
8672

87-
const baseUrl = await getMothershipUrl(environment, userId)
88-
if (!baseUrl) {
89-
return NextResponse.json(
90-
{ error: `No URL configured for environment: ${environment}` },
91-
{ status: 400 }
92-
)
93-
}
73+
/** The upstream's own explanation, from either error envelope the admin API uses. */
74+
function upstreamErrorMessage(data: unknown): string | undefined {
75+
if (!isRecordLike(data)) return undefined
76+
if (typeof data.error === 'string') return data.error
77+
if (typeof data.message === 'string') return data.message
78+
return undefined
79+
}
9480

95-
const targetUrl = `${baseUrl}/api/admin/${endpoint}`
81+
type ProxyMethod = 'GET' | 'POST' | 'DELETE'
9682

83+
/**
84+
* Forwards to the mothership admin API. A 4xx passes through for the admin UI to show; an
85+
* upstream 5xx or unparseable body is a gateway failure, logged with its cause and answered
86+
* with 502 so it is not mistaken for a failure of this route.
87+
*/
88+
async function forwardToMothership(params: {
89+
method: ProxyMethod
90+
targetUrl: string
91+
adminKey: string
92+
environment: string
93+
endpoint: string
94+
body?: string
95+
}) {
96+
const { method, targetUrl, adminKey, environment, endpoint, body } = params
9797
try {
98-
const body = await req.text()
9998
const upstream = await fetch(targetUrl, {
100-
method: 'POST',
99+
method,
101100
headers: {
102-
'Content-Type': 'application/json',
101+
...(method === 'POST' ? { 'Content-Type': 'application/json' } : {}),
103102
'x-api-key': adminKey,
104103
},
105104
...(body ? { body } : {}),
106105
})
106+
const text = await upstream.text()
107+
if (!text && upstream.status < 500) {
108+
return new NextResponse(null, { status: upstream.status })
109+
}
110+
const data = text ? parseJsonText(text) : null
111+
112+
if (upstream.status >= 500 || data === undefined) {
113+
logger.error('Mothership admin API request failed', {
114+
method,
115+
environment,
116+
endpoint,
117+
status: upstream.status,
118+
body: truncate(text, UPSTREAM_BODY_LOG_LIMIT),
119+
})
120+
return NextResponse.json(
121+
{
122+
error:
123+
upstreamErrorMessage(data) ??
124+
`Mothership (${environment}) returned HTTP ${upstream.status}${data === undefined ? ' with a non-JSON body' : ''}`,
125+
},
126+
{ status: 502 }
127+
)
128+
}
107129

108-
const data = await upstream.json()
109130
return NextResponse.json(data, { status: upstream.status })
110131
} catch (error) {
132+
logger.error('Failed to reach mothership admin API', {
133+
method,
134+
environment,
135+
endpoint,
136+
error: getErrorMessage(error, 'Unknown error'),
137+
})
111138
return NextResponse.json(
112139
{
113140
error: `Failed to reach mothership (${environment}): ${getErrorMessage(error, 'Unknown error')}`,
114141
},
115142
{ status: 502 }
116143
)
117144
}
118-
})
145+
}
119146

120-
export const GET = withRouteHandler(async (req: NextRequest) => {
147+
/**
148+
* Query params:
149+
* env - "dev" | "staging" | "prod"
150+
* endpoint - the admin endpoint path, e.g. "requests", "licenses", "traces"
151+
*
152+
* The request body (for POST) is forwarded as-is. For GET and DELETE, additional query
153+
* params (e.g. requestId for GET /traces) are forwarded.
154+
*/
155+
async function proxyAdminRequest(req: NextRequest, method: ProxyMethod) {
121156
const userId = await getAuthorizedAdminUserId()
122157
if (!userId) {
123158
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
124159
}
125160

126161
const adminKey = env.MOTHERSHIP_API_ADMIN_KEY
127162
if (!adminKey) {
163+
logger.error('MOTHERSHIP_API_ADMIN_KEY is not configured', { method })
128164
return NextResponse.json({ error: 'MOTHERSHIP_API_ADMIN_KEY not configured' }, { status: 500 })
129165
}
130166

@@ -146,85 +182,25 @@ export const GET = withRouteHandler(async (req: NextRequest) => {
146182
}
147183

148184
const forwardParams = new URLSearchParams()
149-
searchParams.forEach((value, key) => {
150-
if (key !== 'env' && key !== 'endpoint') {
151-
forwardParams.set(key, value)
152-
}
153-
})
154-
155-
const qs = forwardParams.toString()
156-
const targetUrl = `${baseUrl}/api/admin/${endpoint}${qs ? `?${qs}` : ''}`
157-
158-
try {
159-
const upstream = await fetch(targetUrl, {
160-
method: 'GET',
161-
headers: { 'x-api-key': adminKey },
185+
if (method !== 'POST') {
186+
searchParams.forEach((value, key) => {
187+
if (key !== 'env' && key !== 'endpoint') {
188+
forwardParams.set(key, value)
189+
}
162190
})
163-
164-
const data = await upstream.json()
165-
return NextResponse.json(data, { status: upstream.status })
166-
} catch (error) {
167-
return NextResponse.json(
168-
{
169-
error: `Failed to reach mothership (${environment}): ${getErrorMessage(error, 'Unknown error')}`,
170-
},
171-
{ status: 502 }
172-
)
173-
}
174-
})
175-
176-
export const DELETE = withRouteHandler(async (req: NextRequest) => {
177-
const userId = await getAuthorizedAdminUserId()
178-
if (!userId) {
179-
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 })
180-
}
181-
182-
const adminKey = env.MOTHERSHIP_API_ADMIN_KEY
183-
if (!adminKey) {
184-
return NextResponse.json({ error: 'MOTHERSHIP_API_ADMIN_KEY not configured' }, { status: 500 })
185-
}
186-
187-
const { searchParams } = new URL(req.url)
188-
const queryValidation = adminMothershipQuerySchema.safeParse(searchParamsToObject(searchParams))
189-
if (!queryValidation.success) return validationErrorResponse(queryValidation.error)
190-
const { env: environment, endpoint } = queryValidation.data
191-
192-
if (!isValidEndpoint(endpoint)) {
193-
return NextResponse.json({ error: 'invalid endpoint' }, { status: 400 })
194191
}
195-
196-
const baseUrl = await getMothershipUrl(environment, userId)
197-
if (!baseUrl) {
198-
return NextResponse.json(
199-
{ error: `No URL configured for environment: ${environment}` },
200-
{ status: 400 }
201-
)
202-
}
203-
204-
const forwardParams = new URLSearchParams()
205-
searchParams.forEach((value, key) => {
206-
if (key !== 'env' && key !== 'endpoint') {
207-
forwardParams.set(key, value)
208-
}
209-
})
210-
211192
const qs = forwardParams.toString()
212-
const targetUrl = `${baseUrl}/api/admin/${endpoint}${qs ? `?${qs}` : ''}`
213193

214-
try {
215-
const upstream = await fetch(targetUrl, {
216-
method: 'DELETE',
217-
headers: { 'x-api-key': adminKey },
218-
})
194+
return forwardToMothership({
195+
method,
196+
targetUrl: `${baseUrl}/api/admin/${endpoint}${qs ? `?${qs}` : ''}`,
197+
adminKey,
198+
environment,
199+
endpoint,
200+
...(method === 'POST' ? { body: await req.text() } : {}),
201+
})
202+
}
219203

220-
const data = await upstream.json()
221-
return NextResponse.json(data, { status: upstream.status })
222-
} catch (error) {
223-
return NextResponse.json(
224-
{
225-
error: `Failed to reach mothership (${environment}): ${getErrorMessage(error, 'Unknown error')}`,
226-
},
227-
{ status: 502 }
228-
)
229-
}
230-
})
204+
export const POST = withRouteHandler((req: NextRequest) => proxyAdminRequest(req, 'POST'))
205+
export const GET = withRouteHandler((req: NextRequest) => proxyAdminRequest(req, 'GET'))
206+
export const DELETE = withRouteHandler((req: NextRequest) => proxyAdminRequest(req, 'DELETE'))

‎apps/sim/app/api/guardrails/mask-batch/route.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ export const POST = withRouteHandler(async (request: NextRequest) => {
4545
try {
4646
const startedAt = performance.now()
4747
const masked = await maskPIIBatch(texts, entityTypes, language, customPatterns)
48-
logger.info('Masked PII batch', {
48+
logger.debug('Masked PII batch', {
4949
count: texts.length,
5050
durationMs: Math.round(performance.now() - startedAt),
5151
})

‎apps/sim/app/api/telemetry/route.ts‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import { createLogger } from '@sim/logger'
2+
import { getErrorMessage } from '@sim/utils/errors'
23
import { type NextRequest, NextResponse } from 'next/server'
34
import { telemetryContract } from '@/lib/api/contracts/telemetry'
45
import { parseRequest } from '@/lib/api/server'
@@ -122,7 +123,7 @@ async function forwardToCollector(data: Record<string, unknown>): Promise<boolea
122123
clearTimeout(timeoutId)
123124

124125
if (!response.ok) {
125-
logger.error('Telemetry collector returned error', {
126+
logger.warn('Telemetry collector returned error', {
126127
status: response.status,
127128
statusText: response.statusText,
128129
})
@@ -133,9 +134,11 @@ async function forwardToCollector(data: Record<string, unknown>): Promise<boolea
133134
} catch (fetchError) {
134135
clearTimeout(timeoutId)
135136
if (fetchError instanceof Error && fetchError.name === 'AbortError') {
136-
logger.error('Telemetry request timed out', { endpoint })
137+
logger.warn('Telemetry request timed out', { endpoint })
137138
} else {
138-
logger.error('Failed to send telemetry to collector', fetchError)
139+
logger.warn('Failed to send telemetry to collector', {
140+
error: getErrorMessage(fetchError),
141+
})
139142
}
140143
return false
141144
}

‎apps/sim/background/webhook-execution.ts‎

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -722,17 +722,32 @@ export async function resolveWebhookExecutionProviderConfig<
722722
}
723723
}
724724

725-
async function resolveCredentialAccountUserId(credentialId: string): Promise<string | undefined> {
725+
/**
726+
* The user who owns the OAuth account behind a credential. `accountless` credentials
727+
* (service accounts, managed OAuth such as a custom Slack bot) have no account row by
728+
* design; `missing` means the credential or its account no longer exists.
729+
*/
730+
async function resolveCredentialAccountUserId(
731+
credentialId: string
732+
): Promise<{ status: 'resolved'; userId: string } | { status: 'accountless' | 'missing' }> {
726733
const resolved = await resolveOAuthAccountId(credentialId)
727734
if (!resolved) {
728-
return undefined
735+
return { status: 'missing' }
736+
}
737+
if (
738+
resolved.credentialType === 'service_account' ||
739+
resolved.credentialType === 'managed_oauth'
740+
) {
741+
return { status: 'accountless' }
729742
}
730743
const [credentialRecord] = await db
731744
.select({ userId: account.userId })
732745
.from(account)
733746
.where(eq(account.id, resolved.accountId))
734747
.limit(1)
735-
return credentialRecord?.userId
748+
return credentialRecord
749+
? { status: 'resolved', userId: credentialRecord.userId }
750+
: { status: 'missing' }
736751
}
737752

738753
/**
@@ -873,15 +888,16 @@ async function executeWebhookJobInternal(
873888
workspaceId
874889
)
875890
: loadDeployedWorkflowState(payload.workflowId, workspaceId)
876-
const [workflowData, webhookRows, resolvedCredentialUserId] = await Promise.all([
891+
const [workflowData, webhookRows, credentialAccount] = await Promise.all([
877892
workflowStatePromise,
878893
db.select().from(webhook).where(eq(webhook.id, payload.webhookId)).limit(1),
879894
payload.credentialId
880895
? resolveCredentialAccountUserId(payload.credentialId)
881896
: Promise.resolve(undefined),
882897
])
883-
const credentialAccountUserId = resolvedCredentialUserId
884-
if (payload.credentialId && !credentialAccountUserId) {
898+
const credentialAccountUserId =
899+
credentialAccount?.status === 'resolved' ? credentialAccount.userId : undefined
900+
if (credentialAccount?.status === 'missing') {
885901
logger.warn(
886902
`[${requestId}] Failed to resolve credential account for credential ${payload.credentialId}`
887903
)

‎apps/sim/blocks/custom/server-overlay.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ import { registerBlockOverlayResolver } from '@/blocks/custom/overlay'
55
import type { BlockConfig, BlockIcon } from '@/blocks/types'
66

77
/** A row for the overlay, optionally carrying live-derived Start input fields. */
8-
type CustomBlockOverlayRow = CustomBlockRow & {
8+
export type CustomBlockOverlayRow = CustomBlockRow & {
99
inputFields?: WorkflowInputField[]
1010
/** When `false`, the block resolves but is hidden from the palette (disabled). */
1111
enabled?: boolean

‎apps/sim/executor/dag/builder.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,7 @@ export class DAGBuilder {
9797
// Validate loop and parallel structure
9898
this.validateSubflowStructure(dag)
9999

100-
logger.info('DAG built', {
100+
logger.debug('DAG built', {
101101
totalNodes: dag.nodes.size,
102102
loopCount: dag.loopConfigs.size,
103103
parallelCount: dag.parallelConfigs.size,

0 commit comments

Comments
 (0)