[jwrosewell]

What this changes

This moves the examples, tests, packaging and CI from the old device-detection-data and ip-intelligence-data repositories to the public data repository. It is step 4 of the run book in 51Degrees/data#1 (version 2), and it moves every link to the old Azure blob storage to the data repository's script or the Distributor.

The data repository reaches this repository through cxx, as the cxx/data submodule. Each free data file there is a <file>.link holding the address it is downloaded from, and get-data-files.ps1 and get-data-files.sh at its root download them.

1. Submodules. cxx moves to e866f6d3, the 4.5.103 release on cxx main, which carries the cxx change that adds the data submodule (Add the data repository as a submodule and move the examples, tests and CI to it 51degrees-cxx#96). This repository's own ip-intelligence/ip-intelligence-data submodule is removed from .gitmodules and from the index.
2. Paths. Every cxx/device-detection-data path is now cxx/data/device-detection, in device-detection/ci/fetch-assets.ps1, the device detection on-premise tests, fiftyone_devicedetection/examples/gettingstarted.py and performance.py. Everything that read ip-intelligence/ip-intelligence-data now reads cxx/data/ip-intelligence. That is the translation package's setup.py and resources.py, the on-premise test constants and the IP Intelligence example_utils.py.
3. Examples download a missing file. The two shared helpers do this, the device detection and IP Intelligence example_utils.py, so no example has its own copy. When a file isn't found and <file>.link is in the data folder, the helper prints the command and runs it. On Windows that is powershell -NoProfile -ExecutionPolicy Bypass -File <data>\get-data-files.ps1 <file> -Source examples-51degrees-python, and elsewhere sh <data>/get-data-files.sh -s examples-51degrees-python <file>. Then it looks again. If the file is still missing, the helper prints the command to run and returns None, and the example prints its own message.
4. The device detection helper looks in cxx/data/device-detection first. It used to search the whole tree by file name and take the first match. cxx keeps its old device-detection-data submodule for one release. In CI, and in any clone without Git LFS, the files there are Git LFS pointers with the same names. So the helper now checks cxx/data/device-detection, in the working directory and each folder above it, before that search.
5. Messages and links. The messages that said to update the device-detection-data or ip-intelligence-data submodule now say to get the data submodule with git submodule update --init --recursive and run the example again. The age warnings name the download script. The device detection readme's Git LFS steps are replaced by the script.
6. Azure blob storage. The IP Intelligence web example and the on-premise readme linked 51ddatafiles.blob.core.windows.net. The web example now looks for 51Degrees-LiteV41.ipi, the name the data repository links the Lite file under, so it downloads the file as the other examples do. LITE_DATAFILE_NAME in fiftyone_ipintelligence_shared changes to that name. Nothing else in the repository uses that constant. The readme gives the script, and gives the Distributor's free download https://distributor.51degrees.com/api/v2/free/ipi-lite-v41 for anyone without a checkout.
7. The ASN file in CI. 51Degrees-IPIV4AsnIpiV41.ipi was committed to ip-intelligence-data and is now a link. ip-intelligence/ci/fetch-assets.ps1 downloads it with the data repository's script. device-detection/ci/fetch-assets.ps1 runs the script for its folder after linking in the assets, which the script keeps, as cxx does.
8. CommittedDataFileTests is now AsnDataFileTests. It checks that the downloaded ASN file is present, and fails with the command that gets it if not. Then it checks the engine opens the file. The Distributor always serves the latest ASN file, so the test now fails when the engine can't read the format 51Degrees publishes.
9. ci/packages.json. The two ExtraPaths entries for the removed submodule are removed rather than renamed. A folder inside cxx never shows in this repository's diffs, so cxx/data/ip-intelligence would match nothing. Naming cxx itself would make a cxx update test only IP Intelligence, because Resolve-AffectedProducts counts a path in a package's folders as touching only that package's product and the products that depend on it. So a cxx update releases neither package, just as it already releases neither on-premise package. AGENTS.md now says so.
10. Docs. The root README says how to download the files, and tells existing clones to delete the old folders, which Git leaves on disk. AGENTS.md and the product readmes describe the new layout.

The translations in the built package

setup.py copies the YAML files from ../../cxx/data/ip-intelligence/Translations, relative to itself. It still returns early when that folder is missing, because a build from the source distribution has no cxx folder but already holds the files. It still ignores errors while copying, too. So a wrong path would build a package without the YAML files, and give no error.

The new test PackagedTranslationsTests.test_translations_are_packaged checks that the package itself holds countrycodes.en_GB.yml and countries.de_DE.yml. tox builds a wheel and installs it before the tests run, so in the full build this test reads the installed package. The other translation tests can't show this on their own. When the package holds no YAML files, resources.py looks for cxx/data/ip-intelligence/Translations in up to eight folders above the installed package, and that search can reach the checkout.

At e866f6d3, cxx pins data at e448dcf, a commit on data's main. It has ip-intelligence/Translations/countrycodes.en_GB.yml and 16 files in ip-intelligence/Translations/OSM, which is what setup.py copies.

Merging rc into this branch

rc moved on after this branch was made. It took the regenerated SWIG wrappers (#27) and released them in #30, and the nightly submodule update (#26) moved two submodules. The merge commit fd49236 brings those in. Two paths conflicted, both of them submodules.

Path	This branch	rc	Kept
cxx	f03dbfd6	1bec7b57, the 4.5.102 release	f03dbfd6, then e866f6d3 in the next commit
ip-intelligence/ip-intelligence-data	removed	b8b3d7d4, moved from b0f0b6af by the nightly	removed

• cxx. f03dbfd6 is 9 commits ahead of 1bec7b57 and none behind, with the same src/ tree (e2adedf2). So it has the 4.5.102 sources rc moved to, plus the data submodule this change needs.
• ip-intelligence/ip-intelligence-data. The nightly's move only adds .github/workflows/only-rc-into-main.yml to that repository (compare). This change reads the IP Intelligence files from cxx/data/ip-intelligence instead, so the submodule stays removed.

The rest merged without conflict.

• The SWIG wrappers. This branch was made from the branch of Generate both SWIG wrappers again for 51degrees-cxx 4.5.102 #27, and its tree after those commits (49cea61) is the same as the squash of Generate both SWIG wrappers again for 51degrees-cxx 4.5.102 #27 on rc (4fc6676). So both sides made the same change.
• js_templates takes the nightly's move from d3f96049 to c773ce22, which changes only the workflows and README of javascript-templates (compare).

After the merge, the tree differs from ce3e12f, this branch's last commit before the merge, only in js_templates. The diff against rc is the same 44 files as before.

The cxx release

cxx released the data submodule change to main in its release pull request (51Degrees/51degrees-cxx#103), as e866f6d3, tagged 4.5.103. The commit after the merge, 5e5ba20, moves cxx from f03dbfd6 to e866f6d3. The two have the same src/ tree (e2adedf2), the same .gitmodules, and the same data (e448dcf), device-detection-data and ip-intelligence-data commits. They differ only in cxx's AGENTS.md, README.md, examples/dd/C/Hash/GettingStarted.c and examples/dd/C/Hash/OfflineProcessing.c. This repository reads only cxx/src and cxx/data, so nothing it builds or reads changes. cxx main now has the data folder, so the nightly submodule update, which moves cxx to the head of cxx main, keeps it.

Order

Nothing has to merge first any more.

1. The SWIG wrappers change (Generate both SWIG wrappers again for 51degrees-cxx 4.5.102 #27) is merged into rc and released in Release candidate #30.
2. The cxx change (Add the data repository as a submodule and move the examples, tests and CI to it 51degrees-cxx#96) is released in cxx 4.5.103, and cxx here points at that release.

What was checked

• The tree was searched for device-detection-data, ip-intelligence-data, 51ddatafiles, blob.core.windows, Git LFS and CommittedDataFile. What remains is the README note for old clones and one code comment about LFS pointers. The two .gitattributes files still mark built .so and .pyd files for Git LFS. No tracked file matches them and they have nothing to do with the data, so they are unchanged.
• On ce3e12f, git diff --name-only ce3e12f~1 ce3e12f -- cxx lists cxx, and git diff --name-only ce3e12f~1 ce3e12f -- cxx/data/ip-intelligence lists nothing.
• The cxx commits were compared through the GitHub API. 1bec7b57 (4.5.102), f03dbfd6, cxx rc and e866f6d3 (4.5.103) all have the src/ tree e2adedf2, so the wrappers Generate both SWIG wrappers again for 51degrees-cxx 4.5.102 #27 generated for 4.5.102 still apply. f03dbfd6, cxx rc and e866f6d3 all pin data at e448dcf.
• common-ci clones with --recurse-submodules and then runs git submodule update --init --recursive, in the configure job that fills the asset cache and in every build job. So cxx/data and its scripts are there when fetch-assets.ps1 runs.
• Every changed Python file parses, and both fetch-assets.ps1 scripts parse in PowerShell 7. Line endings are unchanged.
• Nothing was built or tested locally.
• Two full builds were run from a temporary branch, with common-ci's merge step turned off. Each builds and tests every package on Python 3.10 to 3.14 on Ubuntu, Windows and macOS, 15 legs in all.
  • Run 41780893450 built ce3e12f, before the merge, and passed on all 15 legs.
  • Run 41853354039 built fd49236, the merge, and passed on all 15 legs. Every leg had checked out fd49236 before 5e5ba20 was pushed.
• The test counts in run 41853354039 match the nightly build of Update submodules #26 on rc (run 41807348984) suite by suite on all 15 legs, with the same number of tests skipped in each suite. The one difference is the translation package's tests, 26 passed rather than 25, which is the new test_translations_are_packaged.
• 5e5ba20 itself has not had a full build. It changes only the cxx pointer, to a commit with the same src/ tree and data as f03dbfd6.

Still to do

• From a clean git clone --recurse-submodules, running the download script and then each getting-started example, and running each example with its files deleted. This hasn't been tried.
• The ASN file is downloaded in each build job, because it isn't one of common-ci's cached assets yet. The Distributor limits how often one address can download a file, so a job could fail with HTTP 429. Once the common-ci change that gets the free files from the data repository (Get the free data files from the data repository common-ci#300) is in common-ci main, the ASN file can come from the asset cache instead.
• Other open pull requests touch the same files.
  • The native memory change (Let FlowData and pipelines release native memory, and close them in the tests and examples #36) conflicts with this in the IP Intelligence gettingstarted_web/app.py. This changes the error message in main(), and Let FlowData and pipelines release native memory, and close them in the tests and examples #36 puts the two lines after it inside with build_pipeline(data_file):. Whichever merges second keeps both.
  • The CI change (Measure changes from rc and build pull requests on fewer entries #33) lists ip-intelligence/ip-intelligence-data in $NativePaths in ci/select-options.ps1. That path is gone once this merges, and cxx already counts on its own.
  • The AGENTS.md change (Say in AGENTS.md how to get the data, build and run the examples #34) describes both layouts. Its "Without cxx/data" part stops applying once this merges.
  • The package metadata and links change (Point package metadata and package READMEs at the right pages #35) touches 11 of the same files and merges with this without conflict. The callback URL change (Check the callback URL before the JavaScript builder writes it #31) and the location example fix (Ask the country property for its no value message in the location example #32) touch none of them.