Repository navigation
Use the data repository for the examples, tests and CI - #37
Merged
Merged
Conversation
cxx now includes the data repository as its data submodule, so the examples, tests and CI look for the data files in cxx/data/device-detection and cxx/data/ip-intelligence rather than cxx/device-detection-data and cxx/ip-intelligence-data. Each free data file is a link there until the data repository's get-data-files script downloads it. ci/fetch-assets.ps1 links the assets into the new folders, then runs the script, which keeps the linked files and downloads the Asn file. The Asn file was committed in ip-intelligence-data and is now a link like the others. The DataFileHelper classes of the examples run the script for a free file that is missing, logging the command they run, and fail naming the command if the file is still missing. The examples that looked for their files directly now go through the helpers. The free Device Detection files are looked for by folder and name, such as device-detection/51Degrees-LiteV4.1.hash, so a search does not stop at a file of the same name in the old device-detection-data folder, where those files are Git LFS pointers. The translation package copies its YAML files from cxx/data/ip-intelligence/Translations, and the docs replace the Git LFS steps with the data repository's scripts.
The cxx pointer moves to the head of the cxx feature/data-submodule branch, which adds the data repository at cxx/data, so the full build runs against the new layout. The nightly submodule update moves the pointer back to cxx main once that change is released.
rc moved cxx to 1bec7b57 (4.5.102) while this branch moved it to f03dbfd6, the change that adds the data submodule. f03dbfd6 contains 1bec7b57, so the merge keeps f03dbfd6. The package updates rc took in #36 come in unchanged.
LITE_HASH_DATA_FILE_NAME, UA_FILE_NAME and EVIDENCE_FILE_NAME are public in device-detection.shared, so they keep their values. New LITE_HASH_DATA_FILE_PATH, UA_FILE_PATH and EVIDENCE_FILE_PATH hold each file's path in the data repository, and the searches use them, as a search for the path does not stop at a Git LFS pointer of the same name in an old device-detection-data folder. FileFinder matches with Path.endsWith, which compares whole path elements on every system.
This was referenced Nov 4, 2026
cxx e866f6d3 is the release of cxx#96. Its src tree and data submodule are the same as f03dbfd6, which this pull request was built against, so nothing the build uses changes.
jwrosewell
added a commit
that referenced
this pull request
Nov 4, 2026
#37 moves the examples, tests and CI to the data repository in cxx/data, and merges before this. The run steps now give only that layout. The steps and pitfalls for the old data submodules and Git LFS are gone, and so is the choice between two layouts. The examples find a missing free file's link with FileFinder and run the download script for it, with Windows PowerShell on Windows and sh elsewhere, so the steps say so, and the IP Intelligence argument and the Enterprise path are the ones DataFileHelper uses now. The line endings pitfall is gone, because the data repository keeps its shell script LF.
jwrosewell
marked this pull request as ready for review
November 4, 2026 18:07
Merged
jwrosewell
added a commit
that referenced
this pull request
Nov 4, 2026
Releases what has collected on `rc` since #30: - [#37](#37) Use the data repository for the examples, tests and CI - [#41](#41) Say in AGENTS.md how to get the data, build and run the examples - [#42](#42) Point package metadata at this repository and fix stale links - [#39](#39) Describe the rc approach to releases in AGENTS.md - [#40](#40) Check the callback URL before the JavaScript builder writes it - [#44](#44) Close every FlowData, Pipeline and metadata collection, and fail the build when one is left open ## Release note From [#44](#44): - `ComponentMetaData.getProperties()`, `ProfileMetaData.getValues()`, `ProfileMetaData.getValues(String)` and `FiftyOneAspectPropertyMetaData.getValues()` now declare `CloseableIterable` rather than `Iterable`, which is what the on-premise engines have always returned. Close the collection, for example in a try-with-resources statement, so that it releases the engine's data set straight away. Source that assigns the result to an `Iterable` still compiles. Code compiled against the old declared types needs recompiling, and a class that implements one of these interfaces needs its method to declare `CloseableIterable`, so this is a small breaking change. The release stays a patch release. - `PipelineBuilder.buildFromConfiguration` now closes the elements it has already built when building fails. - The servlet filter and the Spring interceptor now close the request's FlowData on every path. ## Notes The cxx submodule moves from [4.5.102](https://github.com/51Degrees/51degrees-cxx/releases/tag/4.5.102) to [4.5.103](https://github.com/51Degrees/51degrees-cxx/releases/tag/4.5.103). That release adds the [data](https://github.com/51Degrees/data) repository at `cxx/data` and changes nothing under `src/`. No commit asks for a major or minor version change, so the publish moves only patch numbers. The last release into `main` was #30, merged at 06:35 UTC on 4 November 2026.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This moves the examples, tests and CI from the two old data submodules of cxx,
cxx/device-detection-dataandcxx/ip-intelligence-data, to thedata repository, which cxx now includes
at
cxx/data. It is the Java part of step 3 of the plan in51Degrees/data#1.
Why
The data repository holds both old data repositories, in the folders
device-detectionandip-intelligence. Each free data file is a<file>.linkholding the address it is downloaded from, and the repository'sget-data-files.ps1andget-data-files.shdownload them. cxx keeps the oldsubmodules for one release, so that each language moves when it is ready, in
one change and with no fallback code.
What changed
cxx/device-detection-datapath is nowcxx/data/device-detection, and everycxx/ip-intelligence-datapath iscxx/data/ip-intelligence. The paths thatFileFindermatches by theirend, such as
device-detection-data/20000 Evidence Records.yml, now startwith
device-detection/orip-intelligence/. This covers the example XMLconfigs,
OnPremDataFileOptionTest, the example classes, bothDataFileHelperclasses andip-intelligence.translation/pom.xml.The
FileUtilstest helper ofdevice-detection.sharedgainsLITE_HASH_DATA_FILE_PATH,UA_FILE_PATHandEVIDENCE_FILE_PATH, eachthe file's name under the
device-detection/folder. Every search thatused the names now uses these, as do the defaults of
GettingStartedOnPremand
MetadataOnPrem. The publishedLITE_HASH_DATA_FILE_NAME,UA_FILE_NAMEandEVIDENCE_FILE_NAMEkeep their values, so code thatuses them is unchanged.
FileFinderreturns the first file whose path endsin the name it is given. CI still checks out the old
device-detection-datainside cxx, where
51Degrees-LiteV4.1.hash,20000 User Agents.csvand20000 Evidence Records.ymlare Git LFS pointers. With the bare names atest such as
EngineTests.Engine_Hash_GetDataSourceTier_Litecould open apointer rather than the file. The licensed
TAC-HashV41.hashandEnterprise-HashV41.hashare still found by name alone, so a file of yourown can be anywhere in the project.
DataFileHelperclasses have a newfindDataFilemethod. When a file ismissing and
<file>.linkis in the data repository, it runspowershell -NoProfile -ExecutionPolicy Bypass -File <data>\get-data-files.ps1 <file> -Source examples-51degrees-javaon Windows, or
sh <data>/get-data-files.sh -s examples-51degrees-java <file>elsewhere, and logs the command. It then looks again, and if the file is
still missing it fails with a message that names the command.
getDataFileLocationandgetEvidenceFileuse it, and so do the examplesthat looked up their files with
FileFinderdirectly. Those are the DeviceDetection
GettingStartedOnPrem,MetadataOnPrem,OfflineProcessingandShareUsageExample, and the Device Detection file ofGettingStartedMixedand
GettingStartedWebMixed.MatchMetricsnow usesgetDataFileLocation(null), which falls back to the Lite file when no datafile is found. The Device Detection on-premise web example looks for the
data file its configuration names before the server starts, so a missing
Lite file is downloaded first.
ci/fetch-assets.ps1links the assets intocxx/data/device-detectionandcxx/data/ip-intelligence, then runscxx/data/get-data-files.ps1. The script keeps a file that is alreadythere, so the linked Enterprise and Lite files stay. It downloads the Asn
file, which was committed in ip-intelligence-data and is now a link. The
job fails if a download fails. The IP Intelligence evidence files are still
generated in the new folder.
ip-intelligence.translation/pom.xmlcopiescountrycodes.en_GB.ymland thecountries.*.ymlfiles fromcxx/data/ip-intelligence/Translations.the doc comments that pointed at device-detection-data or
ip-intelligence-data now point at the data repository and its scripts.
README.mdreplaces the Git LFS steps with the download script, and tellsexisting clones to delete the old folders once cxx no longer includes them.
AGENTS.md,ci/README.md, the Device Detection and IP IntelligenceREADMEs and both example READMEs are updated. The
device-detection-data/rule in
examples/ip-intelligence/.gitignoreis removed, because the mixedexamples now use the file in
cxx/data, which the data repository's own.gitignorecovers.f03dbfd6, the head of the cxxfeature/data-submodule
branch, which adds the data submodule beside the two old ones
(Add the data repository as a submodule and move the examples, tests and CI to it 51degrees-cxx#96). The pointer returns
to cxx
mainthrough the nightly submodule update once that change isreleased.
rcis merged in at70fe7bc0. It had moved cxx to1bec7b57(4.5.102),which
f03dbfd6already contains, so the merge keepsf03dbfd6.How the full build shows the translations are in the jar
The copy step skips a folder that is not there, without an error. The jar is
packed from
target/classes, andTranslationTestsread the files from therethrough the classpath.
countryNamesFromCodesexpects "United Kingdom" fromcountrycodes.en_GB.yml,translatedCountryexpects "Royaume-Uni" fromcountries.fr_FR.yml, andallListsProducedCorrectlySortedexpects all 250countries. Those tests fail if the copy found nothing. The build log also
shows how many files the
copy-translation-datastep copied.What was checked
device-detection-data,ip-intelligence-data,Git LFS and
get-lite-file-from-azure. What remains is the two notes inAGENTS.mdandFileUtils.javaon why the free files are found by folder,and the advice in
README.mdto delete the old folders.e448dcf0, the commit the cxx branchpoints at. The links, the scripts and the
Translationsfiles are wherethese paths expect them.
ci/fetch-assets.ps1parses in PowerShell 7. The new step checks$?after calling the script, because
$LASTEXITCODEcan still hold an oldervalue. A test script confirmed that
$?is false after a script'sexit 1.-Sourceof the download commandas intended, for a name with spaces too. That was checked with a script
that has the same parameters as
get-data-files.ps1.51degrees-java.and has not passed yet.
Still to check
rc.git clone --recurse-submodules, with the data files deleted,the getting-started examples download them.
common-ci's fetch-assets step gets the free files through the data
repository and caches them (Get the free data files from the data repository common-ci#300).