Skip to content

Use the data repository for the examples, tests and CI - #37

Merged
jwrosewell merged 5 commits into
rcfrom
feature/data-submodule
Nov 4, 2026
Merged

jwrosewell merged 5 commits into
rcfrom
feature/data-submodule

Conversation

@jwrosewell

@jwrosewell jwrosewell commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

This moves the examples, tests and CI from the two old data submodules of cxx,
cxx/device-detection-data and cxx/ip-intelligence-data, to the
data repository, which cxx now includes
at cxx/data. It is the Java part of step 3 of the plan in
51Degrees/data#1.

Why

The data repository holds both old data repositories, in the folders
device-detection and ip-intelligence. Each free data file is a
<file>.link holding the address it is downloaded from, and the repository's
get-data-files.ps1 and get-data-files.sh download them. cxx keeps the old
submodules for one release, so that each language moves when it is ready, in
one change and with no fallback code.

What changed

  1. Paths. Every cxx/device-detection-data path is now
    cxx/data/device-detection, and every cxx/ip-intelligence-data path is
    cxx/data/ip-intelligence. The paths that FileFinder matches by their
    end, such as device-detection-data/20000 Evidence Records.yml, now start
    with device-detection/ or ip-intelligence/. This covers the example XML
    configs, OnPremDataFileOptionTest, the example classes, both
    DataFileHelper classes and ip-intelligence.translation/pom.xml.
  2. The free Device Detection files are found by folder and name.
    The FileUtils test helper of device-detection.shared gains
    LITE_HASH_DATA_FILE_PATH, UA_FILE_PATH and EVIDENCE_FILE_PATH, each
    the file's name under the device-detection/ folder. Every search that
    used the names now uses these, as do the defaults of GettingStartedOnPrem
    and MetadataOnPrem. The published LITE_HASH_DATA_FILE_NAME,
    UA_FILE_NAME and EVIDENCE_FILE_NAME keep their values, so code that
    uses them is unchanged. FileFinder returns the first file whose path ends
    in the name it is given. CI still checks out the old device-detection-data
    inside cxx, where 51Degrees-LiteV4.1.hash, 20000 User Agents.csv and
    20000 Evidence Records.yml are Git LFS pointers. With the bare names a
    test such as EngineTests.Engine_Hash_GetDataSourceTier_Lite could open a
    pointer rather than the file. The licensed TAC-HashV41.hash and
    Enterprise-HashV41.hash are still found by name alone, so a file of your
    own can be anywhere in the project.
  3. The examples download a free file that is missing. Both
    DataFileHelper classes have a new findDataFile method. When a file is
    missing and <file>.link is in the data repository, it runs
    powershell -NoProfile -ExecutionPolicy Bypass -File <data>\get-data-files.ps1 <file> -Source examples-51degrees-java
    on Windows, or sh <data>/get-data-files.sh -s examples-51degrees-java <file>
    elsewhere, and logs the command. It then looks again, and if the file is
    still missing it fails with a message that names the command.
    getDataFileLocation and getEvidenceFile use it, and so do the examples
    that looked up their files with FileFinder directly. Those are the Device
    Detection GettingStartedOnPrem, MetadataOnPrem, OfflineProcessing and
    ShareUsageExample, and the Device Detection file of GettingStartedMixed
    and GettingStartedWebMixed. MatchMetrics now uses
    getDataFileLocation(null), which falls back to the Lite file when no data
    file is found. The Device Detection on-premise web example looks for the
    data file its configuration names before the server starts, so a missing
    Lite file is downloaded first.
  4. CI. ci/fetch-assets.ps1 links the assets into
    cxx/data/device-detection and cxx/data/ip-intelligence, then runs
    cxx/data/get-data-files.ps1. The script keeps a file that is already
    there, so the linked Enterprise and Lite files stay. It downloads the Asn
    file, which was committed in ip-intelligence-data and is now a link. The
    job fails if a download fails. The IP Intelligence evidence files are still
    generated in the new folder.
  5. Translations. ip-intelligence.translation/pom.xml copies
    countrycodes.en_GB.yml and the countries.*.yml files from
    cxx/data/ip-intelligence/Translations.
  6. Messages, links and docs. The log messages, the web page warning and
    the doc comments that pointed at device-detection-data or
    ip-intelligence-data now point at the data repository and its scripts.
    README.md replaces the Git LFS steps with the download script, and tells
    existing clones to delete the old folders once cxx no longer includes them.
    AGENTS.md, ci/README.md, the Device Detection and IP Intelligence
    READMEs and both example READMEs are updated. The device-detection-data/
    rule in examples/ip-intelligence/.gitignore is removed, because the mixed
    examples now use the file in cxx/data, which the data repository's own
    .gitignore covers.
  7. The cxx pointer moves to f03dbfd6, the head of the cxx
    feature/data-submodule
    branch, which adds the data submodule beside the two old ones
    (Add the data repository as a submodule and move the examples, tests and CI to it 51degrees-cxx#96). The pointer returns
    to cxx main through the nightly submodule update once that change is
    released.
    rc is merged in at 70fe7bc0. It had moved cxx to 1bec7b57 (4.5.102),
    which f03dbfd6 already contains, so the merge keeps f03dbfd6.

How the full build shows the translations are in the jar

The copy step skips a folder that is not there, without an error. The jar is
packed from target/classes, and TranslationTests read the files from there
through the classpath. countryNamesFromCodes expects "United Kingdom" from
countrycodes.en_GB.yml, translatedCountry expects "Royaume-Uni" from
countries.fr_FR.yml, and allListsProducedCorrectlySorted expects all 250
countries. Those tests fail if the copy found nothing. The build log also
shows how many files the copy-translation-data step copied.

What was checked

  • Every file was searched for device-detection-data, ip-intelligence-data,
    Git LFS and get-lite-file-from-azure. What remains is the two notes in
    AGENTS.md and FileUtils.java on why the free files are found by folder,
    and the advice in README.md to delete the old folders.
  • The data repository's tree was read at e448dcf0, the commit the cxx branch
    points at. The links, the scripts and the Translations files are where
    these paths expect them.
  • ci/fetch-assets.ps1 parses in PowerShell 7. The new step checks $?
    after calling the script, because $LASTEXITCODE can still hold an older
    value. A test script confirmed that $? is false after a script's
    exit 1.
  • Windows PowerShell binds the file name and -Source of the download command
    as intended, for a name with spaces too. That was checked with a script
    that has the same parameters as get-data-files.ps1.
  • The UTM link lint is clean for the campaign 51degrees-java.
  • Each changed file keeps its line endings.
  • Nothing was built or run locally. The full build runs on GitHub's runners
    and has not passed yet.

Still to check

  • The full build passes, with no more tests skipped than on rc.
  • From a clean git clone --recurse-submodules, with the data files deleted,
    the getting-started examples download them.
  • Each CI job downloads the Asn file from the Distributor itself, until
    common-ci's fetch-assets step gets the free files through the data
    repository and caches them (Get the free data files from the data repository common-ci#300).

cxx now includes the data repository as its data submodule, so the
examples, tests and CI look for the data files in cxx/data/device-detection
and cxx/data/ip-intelligence rather than cxx/device-detection-data and
cxx/ip-intelligence-data.

Each free data file is a link there until the data repository's
get-data-files script downloads it. ci/fetch-assets.ps1 links the assets
into the new folders, then runs the script, which keeps the linked files
and downloads the Asn file. The Asn file was committed in
ip-intelligence-data and is now a link like the others.

The DataFileHelper classes of the examples run the script for a free file
that is missing, logging the command they run, and fail naming the command
if the file is still missing. The examples that looked for their files
directly now go through the helpers.

The free Device Detection files are looked for by folder and name, such as
device-detection/51Degrees-LiteV4.1.hash, so a search does not stop at a
file of the same name in the old device-detection-data folder, where those
files are Git LFS pointers.

The translation package copies its YAML files from
cxx/data/ip-intelligence/Translations, and the docs replace the Git LFS
steps with the data repository's scripts.
The cxx pointer moves to the head of the cxx feature/data-submodule
branch, which adds the data repository at cxx/data, so the full build
runs against the new layout. The nightly submodule update moves the
pointer back to cxx main once that change is released.
rc moved cxx to 1bec7b57 (4.5.102) while this branch moved it to f03dbfd6, the
change that adds the data submodule. f03dbfd6 contains 1bec7b57, so the merge
keeps f03dbfd6. The package updates rc took in #36 come in unchanged.
LITE_HASH_DATA_FILE_NAME, UA_FILE_NAME and EVIDENCE_FILE_NAME are public in
device-detection.shared, so they keep their values. New LITE_HASH_DATA_FILE_PATH,
UA_FILE_PATH and EVIDENCE_FILE_PATH hold each file's path in the data
repository, and the searches use them, as a search for the path does not stop
at a Git LFS pointer of the same name in an old device-detection-data folder.
FileFinder matches with Path.endsWith, which compares whole path elements on
every system.
cxx e866f6d3 is the release of cxx#96. Its src tree and data
submodule are the same as f03dbfd6, which this pull request was built
against, so nothing the build uses changes.
jwrosewell added a commit that referenced this pull request Nov 4, 2026
#37 moves the examples, tests and CI to the data repository in cxx/data, and
merges before this. The run steps now give only that layout. The steps and
pitfalls for the old data submodules and Git LFS are gone, and so is the
choice between two layouts.

The examples find a missing free file's link with FileFinder and run the
download script for it, with Windows PowerShell on Windows and sh elsewhere,
so the steps say so, and the IP Intelligence argument and the Enterprise path
are the ones DataFileHelper uses now. The line endings pitfall is gone,
because the data repository keeps its shell script LF.
@jwrosewell
jwrosewell marked this pull request as ready for review November 4, 2026 18:07
@jwrosewell
jwrosewell merged commit 0da326d into rc Nov 4, 2026
2 checks passed
@jwrosewell
jwrosewell deleted the feature/data-submodule branch November 4, 2026 18:07
@jwrosewell jwrosewell mentioned this pull request Nov 4, 2026
jwrosewell added a commit that referenced this pull request Nov 4, 2026
Releases what has collected on `rc` since #30:

- [#37](#37) Use the data repository for the examples, tests and CI
- [#41](#41) Say in AGENTS.md how to get the data, build and run the examples
- [#42](#42) Point package metadata at this repository and fix stale links
- [#39](#39) Describe the rc approach to releases in AGENTS.md
- [#40](#40) Check the callback URL before the JavaScript builder writes it
- [#44](#44) Close every FlowData, Pipeline and metadata collection, and fail the build when one is left open

## Release note

From [#44](#44):

- `ComponentMetaData.getProperties()`, `ProfileMetaData.getValues()`, `ProfileMetaData.getValues(String)` and `FiftyOneAspectPropertyMetaData.getValues()` now declare `CloseableIterable` rather than `Iterable`, which is what the on-premise engines have always returned. Close the collection, for example in a try-with-resources statement, so that it releases the engine's data set straight away. Source that assigns the result to an `Iterable` still compiles. Code compiled against the old declared types needs recompiling, and a class that implements one of these interfaces needs its method to declare `CloseableIterable`, so this is a small breaking change. The release stays a patch release.
- `PipelineBuilder.buildFromConfiguration` now closes the elements it has already built when building fails.
- The servlet filter and the Spring interceptor now close the request's FlowData on every path.

## Notes

The cxx submodule moves from [4.5.102](https://github.com/51Degrees/51degrees-cxx/releases/tag/4.5.102) to [4.5.103](https://github.com/51Degrees/51degrees-cxx/releases/tag/4.5.103). That release adds the [data](https://github.com/51Degrees/data) repository at `cxx/data` and changes nothing under `src/`.

No commit asks for a major or minor version change, so the publish moves only patch numbers.

The last release into `main` was #30, merged at 06:35 UTC on 4 November 2026.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant