fix(lobbying): full pagination, clickable links, lobbyist terminology, fix incomplete client/firm lists - #2239
Draft
nesanders wants to merge 4 commits into
Draft
Conversation
…ata scale
Three related Firestore document/field size-limit bugs surfaced when the
weekly incremental scraper was run for the first time against the full
production-scale dataset (300K+ filings):
1. The live weekly cursor (scrapers/lobbying) stored the entire processed-URL
history and summary cache as two fields on one document. That document
exceeded Firestore's 1MB limit partway through a run, silently failing
(and thus skipping) every registrant processed afterward. Moved to
subcollections — one small doc per URL — mirroring the pattern the
backfill cursor already used, with point lookups instead of an in-memory
set/dict.
2. compute_stats() streamed the full lobbyingFilings/lobbyingRegistrants
collections (300K+ docs) in one unbounded query, which timed out
server-side; the client library's automatic stream-retry then crashed on
an internal AttributeError instead of recovering. Replaced with
cursor-paginated batches (50K docs/request) and a manual retry that
re-issues a fresh query rather than resuming a broken stream.
3. Once (2) was fixed, compute_stats() reached a third limit: the
billSummaries_{court} JSON blob itself exceeded Firestore's 1MB
field-size limit for the current session (1,057KB for court 194's ~5,600
bills), with courts 192/193 close behind. Restructured to one small doc
per bill in a bills subcollection instead of one JSON blob per court —
same fix pattern as (1), applied to writer.py, seedLobbyingStats.ts, and
the frontend fetcher in components/db/lobbying.ts.
All three fixes validated end-to-end against dev Firestore at current scale
(373K filings, 25.6K registrants, 11 courts including the previously-failing
194th).
Also includes scripts/firebase-admin/checkLobbyingFreshness.ts, a read-only
diagnostic for checking scraper cursor state and data recency.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
run_backfill() marked a year "complete" after one pass and skipped it forever on every future run. That's wrong for the current (still-accruing) year: a run partway through the year would mark it complete based on whatever existed at that moment, silently missing every disclosure filed afterward — no future backfill run would ever see it again. This is exactly what happened to 2026 in production: marked complete in July with 0 disclosures captured. run_backfill already has a fully correct, granular completeness check — _is_backfill_processed, a per-URL subcollection lookup. The year-level flag only ever bought a coarse fast-path (skip re-listing a year's registrants entirely) and it's what caused the bug. Removed it: every run now always re-lists every requested year (one cheap HTTP request per year) and relies solely on the per-URL cursor for correctness, so no year can ever be skipped wholesale again. Added tests/test_scrape.py with a small in-memory Firestore fake (real enough to simulate write-then-read-back across calls, unlike a plain mock) covering both cursor systems: - Regression test reproducing the exact bug scenario (empty pass, then real data appears for the same year) — fails against the old code with 4/4 backfill tests red, passes with the fix, confirmed by checking out the pre-fix scrape.py and rerunning the suite against it. - Backfill always re-lists every year, per-URL dedup still works, dry-run never touches Firestore, completedYears is never written anywhere. - Weekly-mode cursor sanity checks (prior-year caching, current-year always live, parent doc stays small — all state in subcollections). Also validated live against dev: --mode backfill --year 2005 --limit 2 run twice confirms the year is re-listed both times while already-processed disclosures are correctly not reprocessed (0 new on both runs, as expected since 2005 was already backfilled in July). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ction
The billSummaries_{court} restructure (previous commit) moved per-bill
counts from a single document field to a `bills` subcollection, but the
existing `match /lobbyingMeta/{id}` rule only covers documents directly in
that collection — Firestore rules aren't recursive, so it never covered the
new subcollection. The bills index page was failing with "Missing or
insufficient permissions" as a result; caught on a preview deployment,
since all earlier validation ran through the Admin SDK, which bypasses
security rules entirely.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…rminology, fix incomplete client/firm lists Four related frontend fixes to the lobbying explorer: 1. Embedded lobbying card (shown on regular MAPLE bill pages) previously capped at 5 filings with a "view all" link out to a separate page. Now shows the full paginated list (10/page) inline, matching the pagination pattern already used elsewhere in the explorer. 2. Client and lobbyist names in the shared LobbyingFilingsTable were plain text everywhere it's used (the embed, and the firm/client detail pages). Now clickable, linking to their respective detail pages — same href pattern already used on the lobbying explorer's own bill detail page. 3. Renamed "Lobbying Firm(s)" to "Lobbyist(s)" throughout the visible UI (stat cards, column headers, page titles, explainer text) — matches the MA Secretary of State's own "Lobbyist Public Search" terminology, which this data is sourced from. Left the /lobbying/firms URL and internal file/variable names unchanged (route stability). 4. Audited data loading across the lobbying feature. Found a real correctness bug, not just a performance one: the clients and firms index pages, and the client detail page's "linked firms" list, all read via useLobbyingAllRegistrants(), which caps at 2,000 of 25,000+ registrant docs (Firestore query limit) — silently showing an incomplete list. Confirmed at the actual data: only 4,815 distinct firms surface via the old registrant scan vs. 7,360 that exist dataset-wide. Fixed by precomputing per-client and per-firm summary docs server-side (writer.py's compute_stats(), mirrored in seedLobbyingStats.ts) over the full, paginated registrants collection — same one-small-doc-per-item subcollection pattern already used for billSummaries, needed here too since ~5,300 clients and ~7,360 firms are already close to Firestore's 1MB single-document limit as flat blobs. Client summary docs also carry a per-firm compensation breakdown, so the client detail page can do a single-document lookup instead of scanning all registrants. Verified live: dev Firestore reseeded, all four changes checked in an actual browser (Playwright against the running dev server) — pagination controls, clickable links, "Lobbyists" terminology, and the corrected 5,135 clients / 7,360 firms counts, zero console errors. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Four fixes to the lobbying explorer:
/lobbying/firmsURL and internal names unchanged.useLobbyingAllRegistrants(), which caps at 2,000 of 25,000+ registrant docs (Firestore query limit) — silently showing an incomplete list. Confirmed against real data: only 4,815 distinct firms surfaced via the old registrant scan vs. 7,360 that actually exist. Fixed by precomputing per-client and per-firm summary docs server-side instead.Checklist
firestore.indexes.json. — N/A, the new queries are unfiltered full-subcollection reads (no composite index needed)Screenshots
Verified locally against the dev Firestore project (seeded via
seedLobbyingStats) with a headless-browser pass over all four changed surfaces — will attach screenshots directly on GitHub before this comes out of draft. In the meantime, here's what each showed:/lobbying/clients/1199SEIU): a "LOBBYISTS" sidebar section listing all firms that represented this client with compensation, all clickable; bills table shows clickable client/lobbyist links./bills/194/H4000): "1–10 of 582" with working Prev/Next controls, clickable client and lobbyist links, "Open in Lobbying Explorer →" link at the bottom.Zero console errors on any of the four pages during verification.
Known issues
compute_stats()/seedLobbyingStatsrun (same as the existingbillSummariesdocs) — fine at current scale (~5,300 clients / ~7,400 firms), but worth watching as the dataset grows.Steps to test/reproduce
yarn firebase-admin run-script seedLobbyingStats --env dev(or wait for the next scrape run) to populate the newlobbyingMeta/clientSummariesandlobbyingMeta/firmSummariessubcollections.firestore.rulesto dev./lobbying/clients— table should show real names, "Lobbyists" (not "Firms") column header, correct total count in the "N clients" line above the table./lobbying/firms— same check; page title/subnav should read "Lobbyists"./bills/194/H4000) and scroll to the lobbying card — should show all filings paginated 10/page (not capped at 5), with clickable client and lobbyist links, and an "Open in Lobbying Explorer →" link at the bottom.