Repository navigation
feat(aws-marketplace): bill a listing's LLM usage to AWS through a per-period metering tally - #9921
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
[Critical impact] The PR appears safe to merge; no new blocking issue was established. SummaryAdds AWS Marketplace billing for billed model usage through per-period tallies and hourly reports.
|
There was a problem hiding this comment.
3 issues found across 13 files
Confidence score: 2/5
route.tsis not scheduled by the Helm chart, so Marketplace usage metering never runs in standard deployments. Add a cron job for this endpoint at the intended 15-minute cadence.0456_aws_marketplace_usage_report.sqlleaves existing organization-linked licenses with aNULLstart time, so the metering query excludes their usage. Backfill those rows to the migration time.metering.tscan drop usage during a mid-hour license handoff: the old license ends at expiry, but the new one starts at the next hour boundary. Adjust the handoff reporting window so usage between those times is included.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/lib/billing/aws-marketplace/metering.ts">
<violation number="1" location="apps/sim/lib/billing/aws-marketplace/metering.ts:184">
P2: A mid-hour license handoff drops usage. When the lapsed license's final row covers only up to its expiry, the next license starts at the following hour boundary, so usage between expiry and that boundary is never billed. Start the successor's chain at the lapsed license's expiry instead of `last.hourStart + HOUR_MS`, or carry the exact boundary forward.</violation>
</file>
<file name="apps/sim/app/api/cron/aws-marketplace-meter/route.ts">
<violation number="1" location="apps/sim/app/api/cron/aws-marketplace-meter/route.ts:25">
P1: This handler is not scheduled by the Helm chart, so Marketplace usage metering never runs in the standard deployment. Add a `cronjobs.jobs` entry for `/api/cron/aws-marketplace-meter` with the intended 15-minute schedule.</violation>
</file>
<file name="packages/db/migrations/0456_aws_marketplace_usage_report.sql">
<violation number="1" location="packages/db/migrations/0456_aws_marketplace_usage_report.sql:19">
P1: Existing organization-linked licenses receive `NULL` here and are then excluded by the metering query, so their organization’s model usage is never billed. Backfill linked rows to the migration time while leaving unlinked rows `NULL`.</violation>
</file>
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
All reported issues were addressed across 13 files
Reply with feedback, questions, or to request a fix.
Turn on auto-fix | Re-trigger cubic
baa1879 to
5dd3ea9
Compare
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
1 issue found across 13 files
Confidence score: 3/5
- Deleting an organization can cascade-delete unsent usage rows in
packages/db/schema.ts, leaving owed usage unbilled even though the license retains its ARN. Preserve these rows or ensure they’re billed before the organization is deleted.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/db/schema.ts">
<violation number="1" location="packages/db/schema.ts:2023">
P2: The `organizationId` FK on `aws_marketplace_usage_report` cascades on organization delete, so deleting an org drops its unsent usage rows. The license keeps its ARN but loses its link, so that owed usage is never billed to AWS. Consider a non-cascading FK, or flushing pending rows before the org is removed.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Turn on auto-fix | Re-trigger cubic
…gh a metering ledger
…iod, so in-place cost top-ups, renewals and cancellations bill exactly
…and keep out-of-window records from failing a batch
…m, flush a license at its expiry, and keep charged usage when its organization is deleted
5dd3ea9 to
60b4fb6
Compare
|
@cubic-dev-ai review this PR |
@waleedlatif1 I have started the AI code review. It will take a few minutes to complete. |
There was a problem hiding this comment.
1 issue found across 13 files
Confidence score: 3/5
metering.tstreatsDuplicateRecordas reported even though it is not accepted usage, which can leave that usage unbilled. Handle it as rejected and preserve its status in the error.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="apps/sim/lib/billing/aws-marketplace/metering.ts">
<violation number="1" location="apps/sim/lib/billing/aws-marketplace/metering.ts:464">
P2: `DuplicateRecord` is not accepted usage, but this branch treats every status except `CustomerNotSubscribed` as reported. Handle `DuplicateRecord` as rejected (and preserve its status in the error) so unbilled usage is not silently marked settled.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Turn on auto-fix | Re-trigger cubic
Summary
platform-with-usagebills the organization's billedmodelusage (BYOK ismodel_unbilledand never billed) asllm_creditsof $0.01;contractbills nothing beyond the contract. Repricing or changing what counts is a plan editusage_logcost is topped up in place (recordCumulativeUsage), so each run re-sums every billing period with a row that can still change (a request only raises its row while it runs, so a row settles after the longest execution timeout plus a day for late callbacks) and charges the whole units above that period's tallyaws_marketplace_usage_tally(organization, dimension, period): cost counted, units charged, and where the count starts. Fractions wait in the tally; units only move forwardaws_marketplace_usage_report(license, dimension, hour), the key AWS deduplicates on: a row gathers units while its hour is open and is sent once it closes, or at once for an ended licenseexpiresAt). A license ends at the earlier of its passed expiry and the check that first finds it inactive, so overlapping licenses never both bill and a renewal handoff needs no partial-hour logic. With none in force, a license that ended within the hour AWS allows takes the usage at the hour it ended, and that record is sent at once (AWS's hour runs from the unsubscribe, which a check only finds later, so later tail usage is deliberately not billed)aws_marketplace_license.ended_at, stamped by the read that first finds a license inactive and cleared if access returns, so a cancellation AWS reports with no entitlement (and no expiry) still gets its final flushaws_marketplace_license.organization_linked_at, stamped wherever a license is linked; usage counts from it. A linked license without one (linked before it existed) is stamped by the first run that sees itBatchMeterUsagebyLicenseArn+CustomerAWSAccountId(noProductCode), 25 per call, grouped per license. Success / DuplicateRecord → reported; CustomerNotSubscribed or a license-level refusal → rejected and logged; unprocessed or transient → pending. Rows AWS won't take (24h old, or last month's after 06:00 UTC on the 1st) are rejected and logged before sending; aTimestampOutOfBoundsExceptionstill raised for a batch is narrowed by sending each row alonesent_at, stamped under the organization's usage lock with the quantity re-read), since AWS keeps the first record it accepts: a retry resends the same quantity, and later usage waits for a record that can still take it/api/cron/aws-marketplace-meterevery 15 minutes (scheduled in infra; SaaS fulfillment only runs on the hosted deployment, likeaws-marketplace-reconcile)organization_idset null), so usage already charged is still sentType of Change
Testing
metering.integration.ts(17, real Postgres; only the AWS metering client is mocked) +lifecycle.integration.tslink-time andended_atcoverage; marketplace integration suites 89/89, unit 10/10ended_atstamp / keep / clear, callers relinking only on change, the charge-side and send-sidesent_atfreeze, out-of-range isolation, month-end cutoff and its grace, the long-run row settle window, expiry as end, expired rows sent at once, report rows surviving org deletionbun run lint,tsc,check:migrations,check:audits(the only local failure is the gitignored docs.contentregistry CI generates), drizzle-kit generate reports no driftChecklist
test-auditauthoring gate)