support pipermail archives - #44
Open
dangowrt wants to merge 2 commits into
Open
Conversation
create_all_tables() migrates an existing lore table before it checks
whether lore_indexed_commits exists. When the lore table predates the
date_timestamp column, migrate_lore_table() calls
reconcile_lore_indexed_commits(), which opens lore_indexed_commits and
fails on databases created before that table was introduced:
Error: Table 'lore_indexed_commits' was not found
Create lore_indexed_commits ahead of the lore migration so the
reconciliation always has a table to open, and add a regression test
that runs create_all_tables() against a database seeded with the old
10-column lore schema.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Projects like U-Boot publish their mailing list archives with pipermail
(Mailman 2) rather than public-inbox, as monthly mbox files linked from
an archive index page. Add a --pipermail option to semcode-index that
downloads and indexes such archives into the existing lore tables:
semcode-index --pipermail https://lists.denx.de/pipermail/u-boot/
The index page is scraped for YYYY-Month.txt(.gz) links and each
monthly file is downloaded to <db_dir>/pipermail/<host>/<list>/, then
split into individual messages and stored through the same insertion
path as lore emails, so lore, dig and vlore work on them unchanged.
Since there is no backing git repository, each message is identified by
a blake3 hash of its content, recorded in lore_indexed_commits for
incremental refreshes.
Pipermail does not escape body lines beginning with "From ", so the
mbox splitter only accepts separator lines with the full From_ shape
(envelope address followed by an asctime date) preceded by a blank
line. This avoids splitting on inline git format-patch headers and on
prose starting with "From ".
--pipermail without arguments refreshes every previously downloaded
archive: only months missing locally and the newest month (which keeps
growing until the next month starts) are fetched again, and message
hashes already recorded are skipped.
A --pipermail-since date (month granularity, accepting the same
formats as the lore search date filters) bounds how far back an
archive is downloaded and indexed. The cutoff is recorded in the
archive directory as archive.since and keeps applying to refreshes;
archives fetched without a cutoff never reach further back than their
oldest already-downloaded month.
The email parsing previously embedded in parse_email_from_commit()
moves to a shared parse_email_content() used by both the lore and the
mbox paths.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
|
Hi @dangowrt! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks! |
meta-cla
Bot
added
the
CLA Signed
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add pipermail archive download support in order to use semcode when working with projects using pipermail which aren't covered by lore.kernel.org (like U-Boot).