Skip to content

feat(aws-marketplace): bill a listing's LLM usage to AWS through a per-period metering tally - #9921

Merged
waleedlatif1 merged 4 commits into
stagingfrom
feat/aws-marketplace-metering
Nov 12, 2026
Merged

waleedlatif1 merged 4 commits into
stagingfrom
feat/aws-marketplace-metering

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Plans carry their usage pricing: platform-with-usage bills the organization's billed model usage (BYOK is model_unbilled and never billed) as llm_credits of $0.01; contract bills nothing beyond the contract. Repricing or changing what counts is a plan edit
  • Metering charges what changed in the books, not hours of rows: usage_log cost is topped up in place (recordCumulativeUsage), so each run re-sums every billing period with a row that can still change (a request only raises its row while it runs, so a row settles after the longest execution timeout plus a day for late callbacks) and charges the whole units above that period's tally
    • aws_marketplace_usage_tally (organization, dimension, period): cost counted, units charged, and where the count starts. Fractions wait in the tally; units only move forward
    • aws_marketplace_usage_report (license, dimension, hour), the key AWS deduplicates on: a row gathers units while its hour is open and is sent once it closes, or at once for an ended license
  • The charged license is the earliest-linked one in force (status active and not past expiresAt). A license ends at the earlier of its passed expiry and the check that first finds it inactive, so overlapping licenses never both bill and a renewal handoff needs no partial-hour logic. With none in force, a license that ended within the hour AWS allows takes the usage at the hour it ended, and that record is sent at once (AWS's hour runs from the unsubscribe, which a check only finds later, so later tail usage is deliberately not billed)
  • aws_marketplace_license.ended_at, stamped by the read that first finds a license inactive and cleared if access returns, so a cancellation AWS reports with no entitlement (and no expiry) still gets its final flush
  • aws_marketplace_license.organization_linked_at, stamped wherever a license is linked; usage counts from it. A linked license without one (linked before it existed) is stamped by the first run that sees it
  • Sending: BatchMeterUsage by LicenseArn + CustomerAWSAccountId (no ProductCode), 25 per call, grouped per license. Success / DuplicateRecord → reported; CustomerNotSubscribed or a license-level refusal → rejected and logged; unprocessed or transient → pending. Rows AWS won't take (24h old, or last month's after 06:00 UTC on the 1st) are rejected and logged before sending; a TimestampOutOfBoundsException still raised for a batch is narrowed by sending each row alone
  • A record's quantity is fixed once it is first sent (sent_at, stamped under the organization's usage lock with the quantity re-read), since AWS keeps the first record it accepts: a retry resends the same quantity, and later usage waits for a record that can still take it
  • Cron /api/cron/aws-marketplace-meter every 15 minutes (scheduled in infra; SaaS fulfillment only runs on the hosted deployment, like aws-marketplace-reconcile)
  • Report rows keep their license on organization deletion (organization_id set null), so usage already charged is still sent
  • Migration 0456 is additive only

Type of Change

  • New feature

Testing

  • metering.integration.ts (17, real Postgres; only the AWS metering client is mocked) + lifecycle.integration.ts link-time and ended_at coverage; marketplace integration suites 89/89, unit 10/10
  • Each guard was removed in turn and its test went red: whole-period re-sum (in-place top-ups), send only after the hour closes, send an ended license at once, stop charging once its end-hour record is sent, skip a license past expiry, stamp unmarked links, period settle window, category filter, count from link, ended_at stamp / keep / clear, callers relinking only on change, the charge-side and send-side sent_at freeze, out-of-range isolation, month-end cutoff and its grace, the long-run row settle window, expiry as end, expired rows sent at once, report rows surviving org deletion
  • bun run lint, tsc, check:migrations, check:audits (the only local failure is the gitignored docs .content registry CI generates), drizzle-kit generate reports no drift

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel

vercel Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
docs Ready Ready Preview Nov 12, 2026 5:18pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Critical impact] The PR appears safe to merge; no new blocking issue was established.

Summary

Adds AWS Marketplace billing for billed model usage through per-period tallies and hourly reports.

  • AWS listings turn eligible model costs into whole usage credits.
  • AWS receives fixed-quantity reports for each billed license hour.
  • A protected cron route starts AWS Marketplace metering runs.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Authenticated cron] --> B[Choose license in force or recently ended]
  B --> C[Re-sum eligible billing periods]
  C --> D[Subtract units already tallied]
  D --> E[Add whole credits to hourly report]
  E --> F{Hour closed or license ended?}
  F -->|No| G[Wait for next run]
  F -->|Yes| H[Lock usage and freeze quantity]
  H --> I[Send to AWS]
  I --> J[Accepted or duplicate: reported]
  I --> K[Temporary failure: retry unchanged]
  I --> L[Permanent refusal: rejected]
Loading

Reviews (4) · Last reviewed commit: "fix(aws-marketplace): count rows while a..." · Reviewed by Greptile

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

3 issues found across 13 files

Confidence score: 2/5

  • route.ts is not scheduled by the Helm chart, so Marketplace usage metering never runs in standard deployments. Add a cron job for this endpoint at the intended 15-minute cadence.
  • 0456_aws_marketplace_usage_report.sql leaves existing organization-linked licenses with a NULL start time, so the metering query excludes their usage. Backfill those rows to the migration time.
  • metering.ts can drop usage during a mid-hour license handoff: the old license ends at expiry, but the new one starts at the next hour boundary. Adjust the handoff reporting window so usage between those times is included.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/billing/aws-marketplace/metering.ts">

<violation number="1" location="apps/sim/lib/billing/aws-marketplace/metering.ts:184">
P2: A mid-hour license handoff drops usage. When the lapsed license's final row covers only up to its expiry, the next license starts at the following hour boundary, so usage between expiry and that boundary is never billed. Start the successor's chain at the lapsed license's expiry instead of `last.hourStart + HOUR_MS`, or carry the exact boundary forward.</violation>
</file>

<file name="apps/sim/app/api/cron/aws-marketplace-meter/route.ts">

<violation number="1" location="apps/sim/app/api/cron/aws-marketplace-meter/route.ts:25">
P1: This handler is not scheduled by the Helm chart, so Marketplace usage metering never runs in the standard deployment. Add a `cronjobs.jobs` entry for `/api/cron/aws-marketplace-meter` with the intended 15-minute schedule.</violation>
</file>

<file name="packages/db/migrations/0456_aws_marketplace_usage_report.sql">

<violation number="1" location="packages/db/migrations/0456_aws_marketplace_usage_report.sql:19">
P1: Existing organization-linked licenses receive `NULL` here and are then excluded by the metering query, so their organization’s model usage is never billed. Backfill linked rows to the migration time while leaving unlinked rows `NULL`.</violation>
</file>

Turn on auto-fix | Re-trigger cubic

@waleedlatif1 waleedlatif1 changed the title feat(aws-marketplace): bill a listing's LLM usage to AWS hourly through a metering ledger feat(aws-marketplace): bill a listing's LLM usage to AWS through a per-period metering tally Nov 12, 2026
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 13 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 force-pushed the feat/aws-marketplace-metering branch from baa1879 to 5dd3ea9 Compare November 12, 2026 06:13
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 13 files

Confidence score: 3/5

  • Deleting an organization can cascade-delete unsent usage rows in packages/db/schema.ts, leaving owed usage unbilled even though the license retains its ARN. Preserve these rows or ensure they’re billed before the organization is deleted.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/db/schema.ts">

<violation number="1" location="packages/db/schema.ts:2023">
P2: The `organizationId` FK on `aws_marketplace_usage_report` cascades on organization delete, so deleting an org drops its unsent usage rows. The license keeps its ARN but loses its link, so that owed usage is never billed to AWS. Consider a non-cascading FK, or flushing pending rows before the org is removed.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread packages/db/schema.ts Resolved
…iod, so in-place cost top-ups, renewals and cancellations bill exactly
…and keep out-of-window records from failing a batch
…m, flush a license at its expiry, and keep charged usage when its organization is deleted
@waleedlatif1
waleedlatif1 force-pushed the feat/aws-marketplace-metering branch from 5dd3ea9 to 60b4fb6 Compare November 12, 2026 17:12
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 12, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 13 files

Confidence score: 3/5

  • metering.ts treats DuplicateRecord as reported even though it is not accepted usage, which can leave that usage unbilled. Handle it as rejected and preserve its status in the error.
Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/billing/aws-marketplace/metering.ts">

<violation number="1" location="apps/sim/lib/billing/aws-marketplace/metering.ts:464">
P2: `DuplicateRecord` is not accepted usage, but this branch treats every status except `CustomerNotSubscribed` as reported. Handle `DuplicateRecord` as rejected (and preserve its status in the error) so unbilled usage is not silently marked settled.</violation>
</file>

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit 3373438 into staging Nov 12, 2026
84 checks passed

This branch was successfully deployed

1 active deployment
Preview — 60b4fb65 Deployed Nov 12, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant