Skip to content

Matching for a specific ecosystem and without matching itself - #905

Open
chinyeungli wants to merge 3 commits into
mainfrom
632_match_maven_jar
Open

Matching for a specific ecosystem and without matching itself#905
chinyeungli wants to merge 3 commits into
mainfrom
632_match_maven_jar

Conversation

@chinyeungli

@chinyeungli chinyeungli commented Aug 25, 2026

Copy link
Copy Markdown
Contributor

Reference: #632

As PurlDB is not a scanner itself, it should not support taking a PURL as input and then performing fetching, extracting, or scanning for matching. Instead, this PR will stick with the current design, where matching is run against the input of a scanned JSON file produced by ScanCode Toolkit or ScanCode.io.

This PR introduces two new fields: ecosystems and exclude_purls.
Screenshot 2026-08-25 114832

With the ecosystems field, matching takes place only within the defined ecosystem. This speeds up the process since it avoids matching against packages from other ecosystems.

With the exclude_purls field, users can specify packages to exclude from matching, preventing a package from being matched against itself.

Sample Inputs

Indexed: pkg:maven/commons-io/commons-io@2.11.0 and pkg:github/nexb/aboutcode-toolkit@v11.1.1
Input: info scan of commons-io and aboutcode-toolkit (commons-io-2.11.0.jar-extract-i.json and aboutcode_toolkit-11.1.1-i.json)

Runs

Verify matching for maven

File: commons-io-2.11.0.jar-extract-i.json
Filters: No

        {
            "url": "http://localhost:8006/api/matching/073896cc-41a7-4822-b93a-1d008867487c/",
            "uuid": "073896cc-41a7-4822-b93a-1d008867487c",
            "created_date": "2026-08-24T03:29:38.562347Z",
            "input_sources": [
                {
                    "filename": "commons-io-2.11.0.jar-extract-i.json",
                    "download_url": "",
                    "is_uploaded": true,
                    "tag": "",
                    "exists": true,
                    "uuid": "bf6c2867-0fe7-444d-9909-5509a143b539"
                }
            ],
            "runs": [
                {
                    "url": "http://localhost:8006/api/runs/1f03ba05-50ae-4ba0-8122-46a35043db37/",
                    "pipeline_name": "matching",
                    "status": "success",
                    "description": "",
                    "project": "http://localhost:8006/api/runs/073896cc-41a7-4822-b93a-1d008867487c/",
                    "uuid": "1f03ba05-50ae-4ba0-8122-46a35043db37",
                    "created_date": "2026-08-24T03:29:38.572943Z",
                    "scancodeio_version": "v7.1.0-97-g076c188",
                    "task_id": "1f03ba05-50ae-4ba0-8122-46a35043db37",
                    "task_start_date": "2026-08-24T03:29:38.612304Z",
                    "task_end_date": "2026-08-24T03:29:57.009797Z",
                    "task_exitcode": 0,
                    "task_output": "",
                    "log": "2026-08-24 03:29:38.616 Pipeline [matching] starting\n2026-08-24 03:29:38.622 Step [download_missing_inputs] starting\n2026-08-24 03:29:38.627 Step [download_missing_inputs] completed in 0 seconds\n2026-08-24 03:29:38.633 Step [get_inputs] starting\n2026-08-24 03:29:38.636 Step [get_inputs] completed in 0 seconds\n2026-08-24 03:29:38.643 Step [build_inventory_from_scans] starting\n2026-08-24 03:29:40.063 Step [build_inventory_from_scans] completed in 1 seconds\n2026-08-24 03:29:40.069 Step [flag_empty_files] starting\n2026-08-24 03:29:40.075 Step [flag_empty_files] completed in 0 seconds\n2026-08-24 03:29:40.081 Step [match_archives_to_purldb_packages] starting\n2026-08-24 03:29:40.088 Skipping resource matching as there are 0\n2026-08-24 03:29:40.098 0 resources matched in PurlDB using 0 SHA1s\n2026-08-24 03:29:40.101 Step [match_archives_to_purldb_packages] completed in 0 seconds\n2026-08-24 03:29:40.107 Step [match_archives_to_purldb_resources] starting\n2026-08-24 03:29:40.113 Skipping resource matching as there are 0\n2026-08-24 03:29:40.121 0 resources matched in PurlDB using 0 SHA1s\n2026-08-24 03:29:40.124 Step [match_archives_to_purldb_resources] completed in 0 seconds\n2026-08-24 03:29:40.131 Step [fingerprint_codebase_directories] starting\n2026-08-24 03:29:40.253 Step [fingerprint_codebase_directories] completed in 0 seconds\n2026-08-24 03:29:40.259 Step [match_directories_exact_to_purldb] starting\n2026-08-24 03:29:40.264 Matching 18 directories against PurlDB\n2026-08-24 03:29:46.282 Progress: 11% (2/18) ETA: 49 seconds\n2026-08-24 03:29:46.288 Progress: 22% (4/18) ETA: 21 seconds\n2026-08-24 03:29:46.363 Progress: 33% (6/18) ETA: 12 seconds\n2026-08-24 03:29:46.369 Progress: 44% (8/18) ETA: 8 seconds\n2026-08-24 03:29:46.375 Progress: 55% (10/18) ETA: 5 seconds\n2026-08-24 03:29:46.381 Progress: 66% (12/18) ETA: 3 seconds\n2026-08-24 03:29:46.386 Progress: 77% (14/18) ETA: 2 seconds\n2026-08-24 03:29:46.393 Progress: 88% (16/18) ETA: 1 seconds\n2026-08-24 03:29:46.398 Progress: 100% (18/18)\n2026-08-24 03:29:46.405 18 directories matched in PurlDB\n2026-08-24 03:29:46.410 Step [match_directories_exact_to_purldb] completed in 6 seconds\n2026-08-24 03:29:46.415 Step [match_directories_to_purldb] starting\n2026-08-24 03:29:46.419 Matching 18 directories against PurlDB\n2026-08-24 03:29:46.427 Progress: 11% (2/18)\n2026-08-24 03:29:46.433 Progress: 22% (4/18)\n2026-08-24 03:29:46.438 Progress: 33% (6/18)\n2026-08-24 03:29:46.445 Progress: 44% (8/18)\n2026-08-24 03:29:46.451 Progress: 55% (10/18)\n2026-08-24 03:29:46.457 Progress: 66% (12/18)\n2026-08-24 03:29:46.463 Progress: 77% (14/18)\n2026-08-24 03:29:46.469 Progress: 88% (16/18)\n2026-08-24 03:29:46.474 Progress: 100% (18/18)\n2026-08-24 03:29:46.480 18 directories matched in PurlDB\n2026-08-24 03:29:46.483 Step [match_directories_to_purldb] completed in 0 seconds\n2026-08-24 03:29:46.489 Step [match_resources_to_purldb] starting\n2026-08-24 03:29:46.493 Matching 206 resources in PurlDB, using SHA1\n2026-08-24 03:29:46.508 Progress: 10% (21/206)\n2026-08-24 03:29:46.512 Progress: 20% (42/206)\n2026-08-24 03:29:46.516 Progress: 30% (62/206)\n2026-08-24 03:29:46.521 Progress: 40% (83/206)\n2026-08-24 03:29:46.524 Progress: 50% (103/206)\n2026-08-24 03:29:46.529 Progress: 60% (124/206)\n2026-08-24 03:29:46.533 Progress: 70% (145/206)\n2026-08-24 03:29:46.537 Progress: 80% (165/206)\n2026-08-24 03:29:46.541 Progress: 90% (186/206)\n2026-08-24 03:29:46.545 Progress: 100% (206/206)\n2026-08-24 03:29:56.917 0 resources matched in PurlDB using 206 SHA1s\n2026-08-24 03:29:56.920 Step [match_resources_to_purldb] completed in 13 seconds\n2026-08-24 03:29:56.925 Step [match_resources_approximately] starting\n2026-08-24 03:29:56.930 Approximate matching 0 resources against PurlDB\n2026-08-24 03:29:56.937 0 resources approximately matched in PurlDB\n2026-08-24 03:29:56.940 Step [match_resources_approximately] completed in 0 seconds\n2026-08-24 03:29:56.945 Step [match_resources_snippets] starting\n2026-08-24 03:29:56.951 Snippet matching 0 resources against PurlDB\n2026-08-24 03:29:56.957 0 resources snippet matched in PurlDB\n2026-08-24 03:29:56.960 Step [match_resources_snippets] completed in 0 seconds\n2026-08-24 03:29:56.965 Step [match_purldb_resources_post_process] starting\n2026-08-24 03:29:56.971 Refining matching for 0 matched-to-purldb-resource archives.\n2026-08-24 03:29:56.976 0 resource processed\n2026-08-24 03:29:56.979 Step [match_purldb_resources_post_process] completed in 0 seconds\n2026-08-24 03:29:56.985 Step [remove_packages_without_resources] starting\n2026-08-24 03:29:56.990 Step [remove_packages_without_resources] completed in 0 seconds\n2026-08-24 03:29:56.995 Pipeline completed in 21 seconds\n",
                    "execution_time": 18
                }
            ],
            "ecosystems_filter": [],
            "exclude_purls_filter": []
        }

Result:

  "path": "org/apache/commons/io/DirectoryWalker.class",
  "type": "file",
  "name": "DirectoryWalker.class",
  "status": "matched-to-purldb-directory",
  "for_packages": [
    "pkg:maven/commons-io/commons-io@2.11.0?uuid=19effd0e-112e-4dc8-8965-eff76eeaeb72"
  ],

Conclusion:
The match success.

Verify exclude purls filter
File: commons-io-2.11.0.jar-extract-i.json
Filters: exclude_purls_filter=pkg:maven/commons-io/commons-io@2.11.0

        {
            "url": "http://localhost:8006/api/matching/7f953c2e-9652-4301-afa6-ce290e5205ee/",
            "uuid": "7f953c2e-9652-4301-afa6-ce290e5205ee",
            "created_date": "2026-08-24T04:23:36.351992Z",
            "input_sources": [
                {
                    "filename": "commons-io-2.11.0.jar-extract-i.json",
                    "download_url": "",
                    "is_uploaded": true,
                    "tag": "",
                    "exists": true,
                    "uuid": "3a67b916-0806-4254-8d7c-57ec7a920710"
                }
            ],
            "runs": [
                {
                    "url": "http://localhost:8006/api/runs/6440302d-ed1b-4895-a548-6f0a73a670cb/",
                    "pipeline_name": "matching",
                    "status": "success",
                    "description": "",
                    "project": "http://localhost:8006/api/runs/7f953c2e-9652-4301-afa6-ce290e5205ee/",
                    "uuid": "6440302d-ed1b-4895-a548-6f0a73a670cb",
                    "created_date": "2026-08-24T04:23:36.358260Z",
                    "scancodeio_version": "v7.1.0-97-g076c188",
                    "task_id": "6440302d-ed1b-4895-a548-6f0a73a670cb",
                    "task_start_date": "2026-08-24T04:23:36.378770Z",
                    "task_end_date": "2026-08-24T04:23:38.394734Z",
                    "task_exitcode": 0,
                    "task_output": "",
                    "log": "2026-08-24 04:23:36.381 Pipeline [matching] starting\n2026-08-24 04:23:36.388 Step [download_missing_inputs] starting\n2026-08-24 04:23:36.392 Step [download_missing_inputs] completed in 0 seconds\n2026-08-24 04:23:36.398 Step [get_inputs] starting\n2026-08-24 04:23:36.401 Step [get_inputs] completed in 0 seconds\n2026-08-24 04:23:36.406 Step [build_inventory_from_scans] starting\n2026-08-24 04:23:37.777 Step [build_inventory_from_scans] completed in 1 seconds\n2026-08-24 04:23:37.783 Step [flag_empty_files] starting\n2026-08-24 04:23:37.788 Step [flag_empty_files] completed in 0 seconds\n2026-08-24 04:23:37.793 Step [match_archives_to_purldb_packages] starting\n2026-08-24 04:23:37.799 Skipping resource matching as there are 0\n2026-08-24 04:23:37.806 0 resources matched in PurlDB using 0 SHA1s\n2026-08-24 04:23:37.809 Step [match_archives_to_purldb_packages] completed in 0 seconds\n2026-08-24 04:23:37.814 Step [match_archives_to_purldb_resources] starting\n2026-08-24 04:23:37.818 Skipping resource matching as there are 0\n2026-08-24 04:23:37.832 0 resources matched in PurlDB using 0 SHA1s\n2026-08-24 04:23:37.838 Step [match_archives_to_purldb_resources] completed in 0 seconds\n2026-08-24 04:23:37.843 Step [fingerprint_codebase_directories] starting\n2026-08-24 04:23:38.004 Step [fingerprint_codebase_directories] completed in 0 seconds\n2026-08-24 04:23:38.010 Step [match_directories_exact_to_purldb] starting\n2026-08-24 04:23:38.015 Matching 18 directories against PurlDB\n2026-08-24 04:23:38.027 Progress: 11% (2/18)\n2026-08-24 04:23:38.038 Progress: 22% (4/18)\n2026-08-24 04:23:38.049 Progress: 33% (6/18)\n2026-08-24 04:23:38.061 Progress: 44% (8/18)\n2026-08-24 04:23:38.073 Progress: 55% (10/18)\n2026-08-24 04:23:38.081 Progress: 66% (12/18)\n2026-08-24 04:23:38.091 Progress: 77% (14/18)\n2026-08-24 04:23:38.100 Progress: 88% (16/18)\n2026-08-24 04:23:38.109 Progress: 100% (18/18)\n2026-08-24 04:23:38.116 0 directories matched in PurlDB\n2026-08-24 04:23:38.119 Step [match_directories_exact_to_purldb] completed in 0 seconds\n2026-08-24 04:23:38.125 Step [match_directories_to_purldb] starting\n2026-08-24 04:23:38.130 Matching 18 directories against PurlDB\n2026-08-24 04:23:38.140 Progress: 11% (2/18)\n2026-08-24 04:23:38.149 Progress: 22% (4/18)\n2026-08-24 04:23:38.159 Progress: 33% (6/18)\n2026-08-24 04:23:38.169 Progress: 44% (8/18)\n2026-08-24 04:23:38.180 Progress: 55% (10/18)\n2026-08-24 04:23:38.188 Progress: 66% (12/18)\n2026-08-24 04:23:38.200 Progress: 77% (14/18)\n2026-08-24 04:23:38.211 Progress: 88% (16/18)\n2026-08-24 04:23:38.221 Progress: 100% (18/18)\n2026-08-24 04:23:38.229 0 directories matched in PurlDB\n2026-08-24 04:23:38.232 Step [match_directories_to_purldb] completed in 0 seconds\n2026-08-24 04:23:38.237 Step [match_resources_to_purldb] starting\n2026-08-24 04:23:38.242 Matching 206 resources in PurlDB, using SHA1\n2026-08-24 04:23:38.251 Progress: 10% (21/206)\n2026-08-24 04:23:38.255 Progress: 20% (42/206)\n2026-08-24 04:23:38.259 Progress: 30% (62/206)\n2026-08-24 04:23:38.263 Progress: 40% (83/206)\n2026-08-24 04:23:38.267 Progress: 50% (103/206)\n2026-08-24 04:23:38.271 Progress: 60% (124/206)\n2026-08-24 04:23:38.275 Progress: 70% (145/206)\n2026-08-24 04:23:38.279 Progress: 80% (165/206)\n2026-08-24 04:23:38.283 Progress: 90% (186/206)\n2026-08-24 04:23:38.287 Progress: 100% (206/206)\n2026-08-24 04:23:38.295 0 resources matched in PurlDB using 206 SHA1s\n2026-08-24 04:23:38.298 Step [match_resources_to_purldb] completed in 0 seconds\n2026-08-24 04:23:38.303 Step [match_resources_approximately] starting\n2026-08-24 04:23:38.308 Approximate matching 5 resources against PurlDB\n2026-08-24 04:23:38.317 0 resources approximately matched in PurlDB\n2026-08-24 04:23:38.319 Step [match_resources_approximately] completed in 0 seconds\n2026-08-24 04:23:38.325 Step [match_resources_snippets] starting\n2026-08-24 04:23:38.330 Snippet matching 5 resources against PurlDB\n2026-08-24 04:23:38.342 0 resources snippet matched in PurlDB\n2026-08-24 04:23:38.346 Step [match_resources_snippets] completed in 0 seconds\n2026-08-24 04:23:38.351 Step [match_purldb_resources_post_process] starting\n2026-08-24 04:23:38.356 Refining matching for 0 matched-to-purldb-resource archives.\n2026-08-24 04:23:38.362 0 resource processed\n2026-08-24 04:23:38.365 Step [match_purldb_resources_post_process] completed in 0 seconds\n2026-08-24 04:23:38.371 Step [remove_packages_without_resources] starting\n2026-08-24 04:23:38.377 Step [remove_packages_without_resources] completed in 0 seconds\n2026-08-24 04:23:38.382 Pipeline completed in 2 seconds\n",
                    "execution_time": 2
                }
            ],
            "ecosystems_filter": [],
            "exclude_purls_filter": [
                "pkg:maven/commons-io/commons-io@2.11.0"
            ]
        },

Result:

  "path": "org/apache/commons/io/DirectoryWalker.class",
  "type": "file",
  "name": "DirectoryWalker.class",
  "status": "",
  "for_packages": [],

Conclusion:
There is no match result because the pkg:maven/commons-io/commons-io@2.11.0 is excluded from the matching.

Verify matching for pypi
File: aboutcode_toolkit-11.1.1-i.json
Filters: No

        {
            "url": "http://localhost:8006/api/matching/5be63a6b-7488-4d8d-8939-6a5526a1a2a6/",
            "uuid": "5be63a6b-7488-4d8d-8939-6a5526a1a2a6",
            "created_date": "2026-08-24T06:12:13.505879Z",
            "input_sources": [
                {
                    "filename": "aboutcode_toolkit-11.1.1-i.json",
                    "download_url": "",
                    "is_uploaded": true,
                    "tag": "",
                    "exists": true,
                    "uuid": "ae361393-53cc-491f-97e6-028a764b3b83"
                }
            ],
            "runs": [
                {
                    "url": "http://localhost:8006/api/runs/742e4b0e-6cf3-427d-a6d7-602ea758159e/",
                    "pipeline_name": "matching",
                    "status": "success",
                    "description": "",
                    "project": "http://localhost:8006/api/runs/5be63a6b-7488-4d8d-8939-6a5526a1a2a6/",
                    "uuid": "742e4b0e-6cf3-427d-a6d7-602ea758159e",
                    "created_date": "2026-08-24T06:12:13.519260Z",
                    "scancodeio_version": "v7.1.0-97-g076c188",
                    "task_id": "742e4b0e-6cf3-427d-a6d7-602ea758159e",
                    "task_start_date": "2026-08-24T06:12:13.548667Z",
                    "task_end_date": "2026-08-24T06:13:09.024306Z",
                    "task_exitcode": 0,
                    "task_output": "",
                    "log": "2026-08-24 06:12:13.553 Pipeline [matching] starting\n2026-08-24 06:12:13.559 Step [download_missing_inputs] starting\n2026-08-24 06:12:13.568 Step [download_missing_inputs] completed in 0 seconds\n2026-08-24 06:12:13.576 Step [get_inputs] starting\n2026-08-24 06:12:13.582 Step [get_inputs] completed in 0 seconds\n2026-08-24 06:12:13.590 Step [build_inventory_from_scans] starting\n2026-08-24 06:12:18.140 Step [build_inventory_from_scans] completed in 5 seconds\n2026-08-24 06:12:18.147 Step [flag_empty_files] starting\n2026-08-24 06:12:18.157 Step [flag_empty_files] completed in 0 seconds\n2026-08-24 06:12:18.164 Step [match_archives_to_purldb_packages] starting\n2026-08-24 06:12:18.171 Matching 6 resources in PurlDB, using SHA1\n2026-08-24 06:12:18.184 0 resources matched in PurlDB using 5 SHA1s\n2026-08-24 06:12:18.188 Step [match_archives_to_purldb_packages] completed in 0 seconds\n2026-08-24 06:12:18.194 Step [match_archives_to_purldb_resources] starting\n2026-08-24 06:12:18.200 Matching 6 resources in PurlDB, using SHA1\n2026-08-24 06:12:18.970 6 resources matched in PurlDB using 5 SHA1s\n2026-08-24 06:12:18.974 Step [match_archives_to_purldb_resources] completed in 1 seconds\n2026-08-24 06:12:18.982 Step [fingerprint_codebase_directories] starting\n2026-08-24 06:12:19.234 Step [fingerprint_codebase_directories] completed in 0 seconds\n2026-08-24 06:12:19.240 Step [match_directories_exact_to_purldb] starting\n2026-08-24 06:12:19.248 Matching 104 directories against PurlDB\n2026-08-24 06:12:19.589 Progress: 10% (11/104) ETA: 3 seconds\n2026-08-24 06:12:19.735 Progress: 20% (21/104) ETA: 2 seconds\n2026-08-24 06:12:19.873 Progress: 30% (32/104) ETA: 1 seconds\n2026-08-24 06:12:20.055 Progress: 40% (42/104) ETA: 1 seconds\n2026-08-24 06:12:20.289 Progress: 50% (52/104) ETA: 1 seconds\n2026-08-24 06:12:20.976 Progress: 60% (63/104) ETA: 1 seconds\n2026-08-24 06:12:21.391 Progress: 70% (73/104) ETA: 1 seconds\n2026-08-24 06:12:21.655 Progress: 80% (84/104) ETA: 1 seconds\n2026-08-24 06:12:22.092 Progress: 90% (94/104)\n2026-08-24 06:12:22.455 Progress: 100% (104/104)\n2026-08-24 06:12:22.575 83 directories matched in PurlDB\n2026-08-24 06:12:22.578 Step [match_directories_exact_to_purldb] completed in 3 seconds\n2026-08-24 06:12:22.584 Step [match_directories_to_purldb] starting\n2026-08-24 06:12:22.590 Matching 104 directories against PurlDB\n2026-08-24 06:12:22.620 Progress: 10% (11/104)\n2026-08-24 06:12:22.646 Progress: 20% (21/104)\n2026-08-24 06:12:22.675 Progress: 30% (32/104)\n2026-08-24 06:12:22.701 Progress: 40% (42/104)\n2026-08-24 06:12:22.731 Progress: 50% (52/104)\n2026-08-24 06:12:22.906 Progress: 60% (63/104)\n2026-08-24 06:12:22.930 Progress: 70% (73/104)\n2026-08-24 06:12:22.957 Progress: 80% (84/104)\n2026-08-24 06:12:22.983 Progress: 90% (94/104)\n2026-08-24 06:12:23.015 Progress: 100% (104/104)\n2026-08-24 06:12:23.027 87 directories matched in PurlDB\n2026-08-24 06:12:23.030 Step [match_directories_to_purldb] completed in 0 seconds\n2026-08-24 06:12:23.036 Step [match_resources_to_purldb] starting\n2026-08-24 06:12:23.042 Matching 399 resources in PurlDB, using SHA1\n2026-08-24 06:12:23.057 Progress: 10% (40/399)\n2026-08-24 06:12:23.063 Progress: 20% (80/399)\n2026-08-24 06:12:23.072 Progress: 30% (120/399)\n2026-08-24 06:12:23.079 Progress: 40% (160/399)\n2026-08-24 06:12:23.084 Progress: 50% (200/399)\n2026-08-24 06:12:23.097 Progress: 60% (240/399)\n2026-08-24 06:12:23.103 Progress: 70% (280/399)\n2026-08-24 06:12:23.108 Progress: 80% (320/399)\n2026-08-24 06:12:23.114 Progress: 90% (360/399)\n2026-08-24 06:12:23.120 Progress: 100% (399/399)\n2026-08-24 06:13:08.893 59 resources matched in PurlDB using 331 SHA1s\n2026-08-24 06:13:08.897 Step [match_resources_to_purldb] completed in 51 seconds\n2026-08-24 06:13:08.902 Step [match_resources_approximately] starting\n2026-08-24 06:13:08.912 Approximate matching 10 resources against PurlDB\n2026-08-24 06:13:08.924 0 resources approximately matched in PurlDB\n2026-08-24 06:13:08.927 Step [match_resources_approximately] completed in 0 seconds\n2026-08-24 06:13:08.934 Step [match_resources_snippets] starting\n2026-08-24 06:13:08.939 Snippet matching 10 resources against PurlDB\n2026-08-24 06:13:08.953 0 resources snippet matched in PurlDB\n2026-08-24 06:13:08.956 Step [match_resources_snippets] completed in 0 seconds\n2026-08-24 06:13:08.962 Step [match_purldb_resources_post_process] starting\n2026-08-24 06:13:08.971 Refining matching for 0 matched-to-purldb-resource archives.\n2026-08-24 06:13:08.977 0 resource processed\n2026-08-24 06:13:08.981 Step [match_purldb_resources_post_process] completed in 0 seconds\n2026-08-24 06:13:08.991 Step [remove_packages_without_resources] starting\n2026-08-24 06:13:08.999 Step [remove_packages_without_resources] completed in 0 seconds\n2026-08-24 06:13:09.007 Pipeline completed in 60 seconds (1.0 minutes)\n",
                    "execution_time": 55
                }
            ],
            "ecosystems_filter": [],
            "exclude_purls_filter": []
        },

Result:

  "path": "tests/testdata/test_gen/lic_key_custom_lic_file/no_lic_key_with_custom_lic_file.csv",
  "type": "file",
  "name": "no_lic_key_with_custom_lic_file.csv",
  "status": "matched-to-purldb-directory",
  "for_packages": [
    "pkg:pypi/aboutcode-toolkit@11.1.1?file_name=aboutcode_toolkit-11.1.1.tar.gz&uuid=cf3477b1-1c12-4457-8873-55aa7d14bb87"
  ],

Conclusion:
The match success.

Verify ecosystem filter
File: aboutcode_toolkit-11.1.1-i.json
Filters: ecosystems_filter=maven

        {
            "url": "http://localhost:8006/api/matching/66ac5462-11ea-4009-b046-8c358cf03834/",
            "uuid": "66ac5462-11ea-4009-b046-8c358cf03834",
            "created_date": "2026-08-24T06:13:56.666719Z",
            "input_sources": [
                {
                    "filename": "aboutcode_toolkit-11.1.1-i.json",
                    "download_url": "",
                    "is_uploaded": true,
                    "tag": "",
                    "exists": true,
                    "uuid": "836468c9-dffd-4893-a482-1df0a7d5bc74"
                }
            ],
            "runs": [
                {
                    "url": "http://localhost:8006/api/runs/b688735d-eac4-4270-b442-790e3cacf78c/",
                    "pipeline_name": "matching",
                    "status": "success",
                    "description": "",
                    "project": "http://localhost:8006/api/runs/66ac5462-11ea-4009-b046-8c358cf03834/",
                    "uuid": "b688735d-eac4-4270-b442-790e3cacf78c",
                    "created_date": "2026-08-24T06:13:56.680398Z",
                    "scancodeio_version": "v7.1.0-97-g076c188",
                    "task_id": "b688735d-eac4-4270-b442-790e3cacf78c",
                    "task_start_date": "2026-08-24T06:13:56.704374Z",
                    "task_end_date": "2026-08-24T06:14:00.480249Z",
                    "task_exitcode": 0,
                    "task_output": "",
                    "log": "2026-08-24 06:13:56.708 Pipeline [matching] starting\n2026-08-24 06:13:56.718 Step [download_missing_inputs] starting\n2026-08-24 06:13:56.722 Step [download_missing_inputs] completed in 0 seconds\n2026-08-24 06:13:56.729 Step [get_inputs] starting\n2026-08-24 06:13:56.731 Step [get_inputs] completed in 0 seconds\n2026-08-24 06:13:56.737 Step [build_inventory_from_scans] starting\n2026-08-24 06:13:58.678 Step [build_inventory_from_scans] completed in 4 seconds\n2026-08-24 06:13:58.684 Step [flag_empty_files] starting\n2026-08-24 06:13:58.698 Step [flag_empty_files] completed in 0 seconds\n2026-08-24 06:13:58.711 Step [match_archives_to_purldb_packages] starting\n2026-08-24 06:13:58.719 Matching 6 resources in PurlDB, using SHA1\n2026-08-24 06:13:58.734 0 resources matched in PurlDB using 5 SHA1s\n2026-08-24 06:13:58.737 Step [match_archives_to_purldb_packages] completed in 0 seconds\n2026-08-24 06:13:58.742 Step [match_archives_to_purldb_resources] starting\n2026-08-24 06:13:58.747 Matching 6 resources in PurlDB, using SHA1\n2026-08-24 06:13:58.763 0 resources matched in PurlDB using 5 SHA1s\n2026-08-24 06:13:58.766 Step [match_archives_to_purldb_resources] completed in 0 seconds\n2026-08-24 06:13:58.777 Step [fingerprint_codebase_directories] starting\n2026-08-24 06:13:59.028 Step [fingerprint_codebase_directories] completed in 0 seconds\n2026-08-24 06:13:59.034 Step [match_directories_exact_to_purldb] starting\n2026-08-24 06:13:59.039 Matching 104 directories against PurlDB\n2026-08-24 06:13:59.081 Progress: 10% (11/104)\n2026-08-24 06:13:59.124 Progress: 20% (21/104)\n2026-08-24 06:13:59.163 Progress: 30% (32/104)\n2026-08-24 06:13:59.203 Progress: 40% (42/104)\n2026-08-24 06:13:59.244 Progress: 50% (52/104)\n2026-08-24 06:13:59.285 Progress: 60% (63/104)\n2026-08-24 06:13:59.333 Progress: 70% (73/104)\n2026-08-24 06:13:59.367 Progress: 80% (84/104)\n2026-08-24 06:13:59.403 Progress: 90% (94/104)\n2026-08-24 06:13:59.436 Progress: 100% (104/104)\n2026-08-24 06:13:59.446 0 directories matched in PurlDB\n2026-08-24 06:13:59.450 Step [match_directories_exact_to_purldb] completed in 0 seconds\n2026-08-24 06:13:59.455 Step [match_directories_to_purldb] starting\n2026-08-24 06:13:59.465 Matching 104 directories against PurlDB\n2026-08-24 06:13:59.509 Progress: 10% (11/104)\n2026-08-24 06:13:59.554 Progress: 20% (21/104)\n2026-08-24 06:13:59.600 Progress: 30% (32/104)\n2026-08-24 06:13:59.644 Progress: 40% (42/104)\n2026-08-24 06:13:59.687 Progress: 50% (52/104)\n2026-08-24 06:13:59.751 Progress: 60% (63/104)\n2026-08-24 06:13:59.802 Progress: 70% (73/104)\n2026-08-24 06:13:59.858 Progress: 80% (84/104)\n2026-08-24 06:13:59.896 Progress: 90% (94/104)\n2026-08-24 06:13:59.940 Progress: 100% (104/104)\n2026-08-24 06:13:59.950 0 directories matched in PurlDB\n2026-08-24 06:13:59.953 Step [match_directories_to_purldb] completed in 0 seconds\n2026-08-24 06:13:59.958 Step [match_resources_to_purldb] starting\n2026-08-24 06:13:59.964 Matching 399 resources in PurlDB, using SHA1\n2026-08-24 06:13:59.984 Progress: 10% (40/399)\n2026-08-24 06:13:59.990 Progress: 20% (80/399)\n2026-08-24 06:13:59.997 Progress: 30% (120/399)\n2026-08-24 06:14:00.002 Progress: 40% (160/399)\n2026-08-24 06:14:00.008 Progress: 50% (200/399)\n2026-08-24 06:14:00.014 Progress: 60% (240/399)\n2026-08-24 06:14:00.020 Progress: 70% (280/399)\n2026-08-24 06:14:00.025 Progress: 80% (320/399)\n2026-08-24 06:14:00.031 Progress: 90% (360/399)\n2026-08-24 06:14:00.037 Progress: 100% (399/399)\n2026-08-24 06:14:00.052 0 resources matched in PurlDB using 331 SHA1s\n2026-08-24 06:14:00.056 Step [match_resources_to_purldb] completed in 0 seconds\n2026-08-24 06:14:00.062 Step [match_resources_approximately] starting\n2026-08-24 06:14:00.067 Approximate matching 447 resources against PurlDB\n2026-08-24 06:14:00.090 Progress: 10% (45/447)\n2026-08-24 06:14:00.103 Progress: 20% (90/447)\n2026-08-24 06:14:00.116 Progress: 30% (135/447)\n2026-08-24 06:14:00.129 Progress: 40% (179/447)\n2026-08-24 06:14:00.143 Progress: 50% (224/447)\n2026-08-24 06:14:00.158 Progress: 60% (269/447)\n2026-08-24 06:14:00.172 Progress: 70% (313/447)\n2026-08-24 06:14:00.189 Progress: 80% (358/447)\n2026-08-24 06:14:00.204 Progress: 90% (403/447)\n2026-08-24 06:14:00.222 Progress: 100% (447/447)\n2026-08-24 06:14:00.234 0 resources approximately matched in PurlDB\n2026-08-24 06:14:00.237 Step [match_resources_approximately] completed in 0 seconds\n2026-08-24 06:14:00.242 Step [match_resources_snippets] starting\n2026-08-24 06:14:00.247 Snippet matching 447 resources against PurlDB\n2026-08-24 06:14:00.269 Progress: 10% (45/447)\n2026-08-24 06:14:00.285 Progress: 20% (90/447)\n2026-08-24 06:14:00.301 Progress: 30% (135/447)\n2026-08-24 06:14:00.315 Progress: 40% (179/447)\n2026-08-24 06:14:00.334 Progress: 50% (224/447)\n2026-08-24 06:14:00.346 Progress: 60% (269/447)\n2026-08-24 06:14:00.358 Progress: 70% (313/447)\n2026-08-24 06:14:00.370 Progress: 80% (358/447)\n2026-08-24 06:14:00.385 Progress: 90% (403/447)\n2026-08-24 06:14:00.405 Progress: 100% (447/447)\n2026-08-24 06:14:00.415 0 resources snippet matched in PurlDB\n2026-08-24 06:14:00.421 Step [match_resources_snippets] completed in 0 seconds\n2026-08-24 06:14:00.427 Step [match_purldb_resources_post_process] starting\n2026-08-24 06:14:00.434 Refining matching for 0 matched-to-purldb-resource archives.\n2026-08-24 06:14:00.442 0 resource processed\n2026-08-24 06:14:00.445 Step [match_purldb_resources_post_process] completed in 0 seconds\n2026-08-24 06:14:00.451 Step [remove_packages_without_resources] starting\n2026-08-24 06:14:00.457 Step [remove_packages_without_resources] completed in 0 seconds\n2026-08-24 06:14:00.464 Pipeline completed in 6 seconds\n",
                    "execution_time": 3
                }
            ],
            "ecosystems_filter": [
                "maven"
            ],
            "exclude_purls_filter": []
        },

Result:

  "path": "tests/testdata/test_gen/lic_key_custom_lic_file/no_lic_key_with_custom_lic_file.csv",
  "type": "file",
  "name": "no_lic_key_with_custom_lic_file.csv",
  "status": "",
  "for_packages": [],

Conclusion:
No match because it strictly limited matching to maven package only.

@chinyeungli
Set DATA_UPLOAD_MAX_MEMORY_SIZE to 200MB #632
70bbce9 
Signed-off-by: Chin Yeung Li <tli@nexb.com>
@chinyeungli
Implement code for matching with maven ecosystem #632
333ceb8 
* Implement the ecosystems (only "maven" at the moment) and exclude_purls fields
* Add tests

Signed-off-by: Chin Yeung Li <tli@nexb.com>
@chinyeungli
Add ecosystems_filter and exclude_purls_filter in the output #632
6b4349b 
Signed-off-by: Chin Yeung Li <tli@nexb.com>
@chinyeungli chinyeungli mentioned this pull request Aug 25, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@chinyeungli