From 9d9d234e8d745c734a1449b1f6a97a54c8ca9902 Mon Sep 17 00:00:00 2001 From: janicelichtman Date: Mon, 16 Nov 2026 16:47:53 -0500 Subject: [PATCH 1/2] adding instructions for ocpf member matching and finance scraping for older courts --- ...pf_member_matching_and_finance_backfill.md | 58 +++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 docs/ocpf_member_matching_and_finance_backfill.md diff --git a/docs/ocpf_member_matching_and_finance_backfill.md b/docs/ocpf_member_matching_and_finance_backfill.md new file mode 100644 index 000000000..60510f53b --- /dev/null +++ b/docs/ocpf_member_matching_and_finance_backfill.md @@ -0,0 +1,58 @@ +# OCPF Member Matching and Finance Scraper Backfill + +This guide explains how to run member matching to connect legislator member IDs with their OCPF IDs and how to run the finance scraper for previous courts. + +## Prerequisites + +- Node.js / yarn installed (`yarn install` to get all deps including `@swc/core`) +- For `dev` / `prod`: a Google service account key, either in `GOOGLE_APPLICATION_CREDENTIALS` or passed via `--creds ` + +## Member matching for previous courts + +The `backfillOcpfMemberMapping` script uses OCPF (Office of Campaign and Political Finance) filers information to look for the OCPF ID for each member of a given court. It stores the matches in Firestore `/generalCourts//ocpfMemberMappingConfig/ocpfMemberMapping` and the members it couldn't match in `/generalCourts//ocpfMemberMappingConfig/ocpfMemberMappingFlags`. The flagged member matches will need to be resolved manually. + +### Running automatic matching + +```sh +# Specify a court +yarn firebase-admin run-script backfillOcpfMemberMapping --env dev --courts specified_court + +# Run for multiple courts +yarn firebase-admin run-script backfillOcpfMemberMapping --env dev --courts court1,court2,court3 # no space + +# To include full details in output, including candidates for ambiguous cases where multiple possible matches were found +yarn firebase-admin run-script backfillOcpfMemberMapping --env dev --courts specified_court --verbose +``` + +### Manual matching + +Cases that could not be automatically matched have to be resolved manually. + +Steps: + +- Look in Firestore for list of members needing manual resolution; ambiguous cases (multiple possible matches) and unmatched cases (no match found): + `/generalCourts//ocpfMemberMappingConfig/ocpfMemberMappingFlags`. +- Find candidate matches for ambiguous cases in output from `--verbose` run of `backfillOcpfMemberMapping`. Some of these will be easy to resolve. (e.g., Tommy Vitolo and Thomas J. Vitolo) +- For ones that are still ambiguous, download the "Filer Database" zip file from OCPF [here](https://ocpf.us/Data/Downloads), and search for the name in all_filers.txt and candidates.txt +- After resolving a match (i.e., have an OCPF ID for a given member ID), run: + `yarn firebase-admin run-script setOcpfMemberMapping --env dev --court court_number --memberCode member_ID --cpfId OCPF_ID` +- Confirm match entered successfully in Firestore: + `/generalCourts//ocpfMemberMappingConfig/ocpfMemberMapping/` + +NOTES: + +- A member of a given court who later switched chambers or changed their last name will come up unmatched or ambiguous, since filers are listed in OCPF with current name and court. + Example: Court 192: Alyson M. Sullivan. Not found in OCPF filers (actually, ambiguous since there are other Sullivans). Not found because by court 193, she changed her name to Alyson M. Sullivan-Almeida. OCPF filers list is one list across time, so a name change (or office change), will mean the person isn’t found for the older court under their original name/office. +- Manually resolved matches are marked as such in Firestore `ocpfMemberMapping`, and will be preserved upon future runs of `backfillOcpfMemberMapping`. + +## Finance scraper for previous courts + +`backfillOcpfFinance` gets campaign finance data for the legislators of a given court, and saves it to Firestore, `/generalCourts//membersFinance` + +```sh +# Specify court number +yarn firebase-admin run-script backfillOcpfFinance --env dev --courts specified_court + +# Run for multiple courts +yarn firebase-admin run-script backfillOcpfFinance --env dev --courts court1,court2,court3 # no space +``` From 12aa969ff99151dcbfe2c3b45d50cc828640c28d Mon Sep 17 00:00:00 2001 From: janicelichtman Date: Mon, 16 Nov 2026 17:09:09 -0500 Subject: [PATCH 2/2] small fixes --- docs/ocpf_member_matching_and_finance_backfill.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/ocpf_member_matching_and_finance_backfill.md b/docs/ocpf_member_matching_and_finance_backfill.md index 60510f53b..8262c9791 100644 --- a/docs/ocpf_member_matching_and_finance_backfill.md +++ b/docs/ocpf_member_matching_and_finance_backfill.md @@ -1,15 +1,16 @@ # OCPF Member Matching and Finance Scraper Backfill -This guide explains how to run member matching to connect legislator member IDs with their OCPF IDs and how to run the finance scraper for previous courts. +This guide explains how to run member matching to connect legislator member IDs with their OCPF IDs, and how to run the finance scraper for previous courts. ## Prerequisites - Node.js / yarn installed (`yarn install` to get all deps including `@swc/core`) +- For local: Firebase emulators running (yarn emulators:start) - For `dev` / `prod`: a Google service account key, either in `GOOGLE_APPLICATION_CREDENTIALS` or passed via `--creds ` ## Member matching for previous courts -The `backfillOcpfMemberMapping` script uses OCPF (Office of Campaign and Political Finance) filers information to look for the OCPF ID for each member of a given court. It stores the matches in Firestore `/generalCourts//ocpfMemberMappingConfig/ocpfMemberMapping` and the members it couldn't match in `/generalCourts//ocpfMemberMappingConfig/ocpfMemberMappingFlags`. The flagged member matches will need to be resolved manually. +The `backfillOcpfMemberMapping` script uses OCPF (Office of Campaign and Political Finance) filers information to look for the OCPF ID for each member of a given court. It stores the matches in Firestore, `/generalCourts//ocpfMemberMappingConfig/ocpfMemberMapping`, and the members it couldn't match in `/generalCourts//ocpfMemberMappingConfig/ocpfMemberMappingFlags`. The flagged member matches need to be resolved manually. ### Running automatic matching @@ -35,19 +36,21 @@ Steps: - Find candidate matches for ambiguous cases in output from `--verbose` run of `backfillOcpfMemberMapping`. Some of these will be easy to resolve. (e.g., Tommy Vitolo and Thomas J. Vitolo) - For ones that are still ambiguous, download the "Filer Database" zip file from OCPF [here](https://ocpf.us/Data/Downloads), and search for the name in all_filers.txt and candidates.txt - After resolving a match (i.e., have an OCPF ID for a given member ID), run: - `yarn firebase-admin run-script setOcpfMemberMapping --env dev --court court_number --memberCode member_ID --cpfId OCPF_ID` + ```sh + yarn firebase-admin run-script setOcpfMemberMapping --env dev --court court_number --memberCode member_ID --cpfId OCPF_ID + ``` - Confirm match entered successfully in Firestore: `/generalCourts//ocpfMemberMappingConfig/ocpfMemberMapping/` NOTES: - A member of a given court who later switched chambers or changed their last name will come up unmatched or ambiguous, since filers are listed in OCPF with current name and court. - Example: Court 192: Alyson M. Sullivan. Not found in OCPF filers (actually, ambiguous since there are other Sullivans). Not found because by court 193, she changed her name to Alyson M. Sullivan-Almeida. OCPF filers list is one list across time, so a name change (or office change), will mean the person isn’t found for the older court under their original name/office. + Example: Court 192: Alyson M. Sullivan. Not found in OCPF filers (actually, ambiguous since there are other Sullivans) because by court 193, she changed her name to Alyson M. Sullivan-Almeida. OCPF filers list is one list across time, so a name change (or office change), will mean the person isn’t found for the older court under their original name/office. - Manually resolved matches are marked as such in Firestore `ocpfMemberMapping`, and will be preserved upon future runs of `backfillOcpfMemberMapping`. ## Finance scraper for previous courts -`backfillOcpfFinance` gets campaign finance data for the legislators of a given court, and saves it to Firestore, `/generalCourts//membersFinance` +`backfillOcpfFinance` gets campaign finance data for the legislators of a given court from OCPF, and saves it to Firestore, `/generalCourts//membersFinance` ```sh # Specify court number