Add script to extract required phrases from rule files

[Kaushik-Kumar-CEG]

references #5077

script that extracts annotated rules (those with {{ }} markers) into a BIOES-labeled JSONL dataset for NER model training

What it does:

• Reads all .RULE files, skips is_required_phrase rules
• Tokenizes using scancode's required_phrase_splitter and generates BIOES labels from markers
• Splits 80/10/10 by license expression to prevent data leakage (heavy expressions split per-rule via hash, light ones kept together)
• Outputs train.jsonl, val.jsonl, test.jsonl

Usage:

python etc/scripts/dataset_pipeline/build_dataset.py --output-dir dataset-output

Tasks

• Reviewed contribution guidelines
• PR is descriptively titled and links the original issue above
• Tests pass
• Commits are in uniquely-named feature branch and has no merge conflicts

disclosure : used Claude to help review and clean up few bugs in script

[Kaushik-Kumar-CEG]

@AyanSinhaMahapatra ready for review when you get a chance. ill make the follow up changes to script as well

[pombredanne]

@Kaushik-Kumar-CEG why not reusing existing code for that https://github.com/aboutcode-org/scancode-toolkit/tree/develop/src/licensedcode like in

scancode-toolkit/src/licensedcode/required_phrases.py

Line 218 in ea42c1d

def collect_is_required_phrase_from_rules(rules_by_expression, verbose=False):

? this script feels redundant work, and it also is missing tests.

[Kaushik-Kumar-CEG]

thanks for the review @pombredanne

collect_is_required_phrase_from_rules is for rules with is_required_phrase: yes which is a different case from what this script handles (inline {{ }} phrases which is what the NER model trains on)

skipping is_required_phrase: yes rules is intentional. those rules already function as required phrases and don't need {{ }} annotation, so they're not in scope for what the model predicts at inference. also tested it in my prototype: including them in training made the model mark entire rule texts as the phrase instead of finding the actual span

on the redundancy, you're right. dropping the local regex in the next commit and using get_required_phrase_verbatim for phrase extraction.for BIOES labels in a later commit, required_phrase_tokenizer will be reused as well

Ill add the required tests once feature commits lands. thanks :)