Skip to content

feat: extract feed_info.txt content - #1796

Open
cka-y wants to merge 4 commits into
mainfrom
feat/1775
Open

feat: extract feed_info.txt content#1796
cka-y wants to merge 4 commits into
mainfrom
feat/1775

Conversation

@cka-y

@cka-y cka-y commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary:
Closes #1775
Creates the architecture to extract file content from GTFS files, plus the first extractor: feed_info.txt. A new gtfs-file-data-extractor function receives one file per invocation and delegates to an extractor resolved by file name, so supporting a new file means adding a FileDataExtractor subclass, registering it in extractors/registry.py, and adding the file name to EXTRACTABLE_FILES in pipeline_tasks.py.
Adds a feedinfo table (one row per dataset, FK to gtfsdataset with cascade delete). Additive only — no API spec or existing column changes.

Expected behavior:
After the daily batch processing (batch-process-dataset), we have a list of files registered for content extraction. For each file we trigger the extraction providing the file url and dataset and feed information. The data is extracted and saved to the database.
Tasks are only enqueued for files whose content changed since the previous dataset, and extraction upserts on the dataset, so reprocessing updates in place instead of duplicating.

Testing tips:
Function can be tested by running the batch processing function or by calling the newly created gtfs-file-data-extractor function directly through HTTP after building this branch on DEV/QA. E.g. of running it in QA:

curl -X POST "https://northamerica-northeast1-mobility-feeds-qa.cloudfunctions.net/gtfs-file-data-extractor" \
-H "Authorization: bearer $(gcloud auth print-identity-token)" \
-H "Content-Type: application/json" \
-d '{
  "stable_id": "mdb-1884",
  "dataset_id": "mdb-1884-202508220038",
  "file_name": "feed_info.txt",
  "file_url": "https://storage.googleapis.com/mobilitydata-datasets-prod/mdb-1884/mdb-1884-202508220038/extracted/feed_info.txt"
}'

Note for deploying: api-deployer creates the function, datasets-batch-deployer creates the task queue. Run api-deployer first, otherwise tasks are enqueued against a function that doesn't exist yet.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@cka-y cka-y changed the title Feat/1775 feat: extract feed_info.txt content Aug 6, 2026
@cka-y
cka-y marked this pull request as ready for review August 6, 2026 16:15
@ianktc
ianktc requested a review from jcpitre August 10, 2026 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Expose feed start and end date from validation report in our DB and/or API

1 participant