Use the data repository for the examples, tests and CI

AGENTS.md

@@ -22,9 +22,18 @@ Tests and examples find data files with `FileFinder`, which searches below
 `project.root`. The surefire configuration sets that to
 `maven.multiModuleProjectDirectory`, which Maven takes from the nearest `.mvn`
 folder above the working directory. The only `.mvn` folder is at the root, so
-the search covers `cxx/device-detection-data` and `cxx/ip-intelligence-data`
+the search covers `cxx/data/device-detection` and `cxx/data/ip-intelligence`
 from wherever Maven is run. Do not add a `.mvn` folder below the root.
 
+Those folders are the data repository, a submodule of cxx, where each free
+data file is a `<file>.link` until `get-data-files.ps1` or `get-data-files.sh`
+downloads it. `ci/fetch-assets.ps1` runs the script, and the examples'
+`DataFileHelper.findDataFile` runs it for a file that is missing. Look for a
+free file by its folder and name, such as
+`device-detection/51Degrees-LiteV4.1.hash`, so the search does not stop at a
+file of the same name elsewhere, such as a Git LFS pointer in an old
+`cxx/device-detection-data` folder.
+
 ## Native builds
 
 Each on-premise engine compiles its own copy of the common C code from `cxx`,

README.md

@@ -44,7 +44,7 @@ Created from `pipeline-java`, `device-detection-java`, `ip-intelligence-java`, `
 | `location` | The Location packages, `com.51degrees:geo-location*` |
 | `examples/device-detection` | Console and web examples for Device Detection |
 | `examples/ip-intelligence` | Console and web examples for IP Intelligence, including the mixed examples that run both products |
-| `cxx` | Submodule of [51degrees-cxx](https://github.com/51Degrees/51degrees-cxx), with the C and C++ code the on-premise engines are built from and the data file submodules |
+| `cxx` | Submodule of [51degrees-cxx](https://github.com/51Degrees/51degrees-cxx), with the C and C++ code the on-premise engines are built from and the [data](https://github.com/51Degrees/data) submodule in `cxx/data` |
 | `pom.xml` | The reactor that builds everything here. It is never published |
 | `ci` | The scripts the shared 51Degrees CI workflows call |
 
@@ -55,10 +55,20 @@ git clone --recurse-submodules https://github.com/51Degrees/51degrees-java.git
 cd 51degrees-java
 ```
 
-The free Device Detection Lite data file in `cxx/device-detection-data` is
-stored with [Git LFS](https://git-lfs.github.com/). Run `git lfs pull` in that
-folder to fetch it. The IP Intelligence ASN file in `cxx/ip-intelligence-data`
-is plain git.
+The free data files come from the [data](https://github.com/51Degrees/data)
+repository, which is the `cxx/data` submodule. Each one is listed there as a
+`.link` file, and a script downloads them all into place:
+
+```
+cxx/data/get-data-files.sh
+```
+
+On Windows, run `cxx\data\get-data-files.ps1` in PowerShell instead. The
+examples run the script themselves for a free file they need that is missing.
+
+In a clone made before the data repository was added, delete the
+`cxx/device-detection-data` and `cxx/ip-intelligence-data` folders once cxx no
+longer includes them, because Git leaves a removed submodule's folder on disk.
 
 ## Building
 
@@ -89,8 +99,9 @@ mvn surefire:test
 ```
 
 The cloud tests need a Resource Key, passed as `-DTestResourceKey=<key>`. The
-on-premise IP Intelligence tests need the Lite or Enterprise data file, which
-is not in the repository. See `ip-intelligence/README.md` for how to get it.
+on-premise tests need the free data files, which the script in
+[Getting the code](#getting-the-code) downloads. For the Enterprise IP
+Intelligence data file, see `ip-intelligence/README.md`.
 Tests find data files by searching from the root of this repository, so run
 Maven from here rather than from inside a package folder.
 

ci/README.md

@@ -21,8 +21,12 @@ repositories used to do for one package tree each.
   workflow instead tests the examples against the version it built, through
   [run-integration-tests.ps1](run-integration-tests.ps1).
 - [fetch-assets.ps1](fetch-assets.ps1) links every data file it downloads into
-  `cxx/device-detection-data` or `cxx/ip-intelligence-data`, where the tests
-  and examples find them.
+  `cxx/data/device-detection` or `cxx/data/ip-intelligence`, where the tests
+  and examples find them. Those are the folders of the
+  [data](https://github.com/51Degrees/data) repository, which is a submodule
+  of cxx. The script then runs the data repository's `get-data-files.ps1`,
+  which keeps the linked files and downloads the free files that are not
+  assets, which is the Asn file.
 - [run-performance-tests.ps1](run-performance-tests.ps1) measures the Pipeline
   API, Device Detection and IP Intelligence and writes one results file per
   configuration. The Device Detection and IP Intelligence metrics are prefixed

ci/fetch-assets.ps1

@@ -7,9 +7,11 @@ param (
 $ErrorActionPreference = "Stop"
 
 # Tests and examples find data files by searching below the repository root,
-# so every asset is linked into the data folders of the cxx submodule.
-$deviceDetectionData = "$PSScriptRoot/../cxx/device-detection-data"
-$ipIntelligenceData = "$PSScriptRoot/../cxx/ip-intelligence-data"
+# so every asset is linked into the folders of the data repository, which is
+# the data submodule of cxx.
+$data = "$PSScriptRoot/../cxx/data"
+$deviceDetectionData = "$data/device-detection"
+$ipIntelligenceData = "$data/ip-intelligence"
 
 $deviceDetectionAssets = 'TAC-HashV41.hash', '51Degrees-LiteV4.1.hash', '20000 Evidence Records.yml', '20000 User Agents.csv'
 ./steps/fetch-assets.ps1 -DeviceDetection:$DeviceDetection -DeviceDetectionUrl:$DeviceDetectionUrl -Assets $deviceDetectionAssets
@@ -24,6 +26,14 @@ New-Item -ItemType SymbolicLink -Force -Target "$PWD/assets/51Degrees-Enterprise
 # link it as that.
 New-Item -ItemType SymbolicLink -Force -Target "$PWD/assets/51Degrees-LiteIpiV41.ipi" -Path "$ipIntelligenceData/51Degrees-LiteV41.ipi"
 
+# The data repository's script downloads every linked file that is not
+# already there. The files linked above are kept, so this gets the free files
+# that are not assets, which is the Asn file.
+& "$data/get-data-files.ps1"
+if (-not $?) {
+    throw "The data repository's get-data-files.ps1 could not download every file"
+}
+
 Write-Host "Assets hashes:"
 Get-FileHash -Algorithm MD5 -Path assets/*
 

device-detection/README.md

@@ -28,23 +28,24 @@ how to use this.
 
 #### On-Premise
 
-If you are using on-premise detection, a "Lite" version of the data required is packaged 
-in this repository. It contains only a limited set of "essential" device detection properties. 
+If you are using on-premise detection, a free "Lite" version of the data required is
+available from the [data](https://github.com/51Degrees/data) repository, which is the
+`cxx/data` folder of this repository. It contains only a limited set of "essential"
+device detection properties.
 
 You may want to license our complete data file containing all properties. 
 [Details of our licenses](https://51degrees.com/pricing?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=on-premise) are available on our website.
 
-If you want to use the lite file, you will need to install [GitLFS](https://git-lfs.github.com/), then:
+To get the Lite file, run the data repository's download script from the root of
+this repository:
 
 ```
-git lfs install
+cxx/data/get-data-files.sh 51Degrees-LiteV4.1.hash
 ```
 
-Then, navigate to 'device-detection.hash.engine.on-premise/src/main/cxx/device-detection-cxx/device-detection-data' and execute:
-
-```
-git lfs pull
-```
+On Windows, run `cxx\data\get-data-files.ps1 51Degrees-LiteV4.1.hash` in PowerShell
+instead. The file is saved in `cxx/data/device-detection`, where the tests and
+examples find it.
 
 ## Installation
 

device-detection/device-detection.cloud/src/test/java/fiftyone/devicedetection/cloud/TestsBase.java

@@ -25,7 +25,7 @@
 import fiftyone.devicedetection.shared.testhelpers.UserAgentGenerator;
 import fiftyone.pipeline.util.FileFinder;
 
-import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_NAME;
+import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_PATH;
 
 public class TestsBase {
 
@@ -43,7 +43,7 @@ protected UserAgentGenerator getUserAgents() {
     protected void testInitialize() throws Exception {
         wrapper = new WrapperCloud();
         userAgents = new UserAgentGenerator(
-            FileFinder.getFilePath(UA_FILE_NAME));
+            FileFinder.getFilePath(UA_FILE_PATH));
     }
 
     public void testCleanup() {

device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/TestsBase.java

@@ -26,7 +26,7 @@
 import fiftyone.devicedetection.shared.testhelpers.FileUtils;
 import fiftyone.pipeline.engines.Constants;
 
-import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_NAME;
+import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_PATH;
 import static fiftyone.pipeline.util.FileFinder.getFilePath;
 
 public class TestsBase {
@@ -47,7 +47,7 @@ protected void testInitialize(Constants.PerformanceProfiles profile) throws Exce
             FileUtils.getHashFile(),
             profile);
         userAgents = new UserAgentGenerator(
-            getFilePath(UA_FILE_NAME));
+            getFilePath(UA_FILE_PATH));
     }
 
     public void testCleanup() {

device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/flowelements/EngineTests.java

@@ -33,7 +33,7 @@
 
 import fiftyone.devicedetection.shared.testhelpers.FileUtils;
 import fiftyone.pipeline.engines.Constants;
-import static fiftyone.devicedetection.shared.testhelpers.FileUtils.LITE_HASH_DATA_FILE_NAME;
+import static fiftyone.devicedetection.shared.testhelpers.FileUtils.LITE_HASH_DATA_FILE_PATH;
 import static fiftyone.devicedetection.shared.testhelpers.FileUtils.TAC_HASH_DATA_FILE_NAME;
 import static fiftyone.pipeline.util.FileFinder.getFilePath;
 
@@ -66,7 +66,7 @@ private void testDataSourceTier(String fileName) throws Exception {
 		String tier = engine.getDataSourceTier();
 		engine.close();
 
-		if (fileName.equals(LITE_HASH_DATA_FILE_NAME)) {
+		if (fileName.equals(LITE_HASH_DATA_FILE_PATH)) {
 			assertEquals("Lite", tier);
 		}
 		else {
@@ -81,7 +81,7 @@ private void testDataSourceTier(String fileName) throws Exception {
 	 */
 	@Test
 	public void Engine_Hash_GetDataSourceTier_Lite() throws Exception {
-		testDataSourceTier(LITE_HASH_DATA_FILE_NAME);
+		testDataSourceTier(LITE_HASH_DATA_FILE_PATH);
 	}
 
 	/**

device-detection/device-detection.shared/src/main/java/fiftyone/devicedetection/shared/testhelpers/FileUtils.java

@@ -47,6 +47,14 @@ public class FileUtils {
     private static File UA_FILE;
     public static final String EVIDENCE_FILE_NAME = "20000 Evidence Records.yml";
     private static Optional<File> EVIDENCE_FILE;
+    // Where each free file is in the data repository. A search for the path
+    // rather than the name does not stop at another file of the same name,
+    // such as a Git LFS pointer in an old device-detection-data folder.
+    public static final String LITE_HASH_DATA_FILE_PATH =
+        "device-detection/" + LITE_HASH_DATA_FILE_NAME;
+    public static final String UA_FILE_PATH = "device-detection/" + UA_FILE_NAME;
+    public static final String EVIDENCE_FILE_PATH =
+        "device-detection/" + EVIDENCE_FILE_NAME;
 
     /**
      * Helper to find the location of an Enterprise or Lite Hash file in the default search scope
@@ -78,7 +86,7 @@ public static File getHashFile() {
             } catch (Exception ex) {
                 try {
                     // following throws exception if not found
-                    HASH_DATA_FILE = Optional.of(getFilePath(LITE_HASH_DATA_FILE_NAME));
+                    HASH_DATA_FILE = Optional.of(getFilePath(LITE_HASH_DATA_FILE_PATH));
                     return HASH_DATA_FILE.get();
                 } catch (Exception exc) {
                     HASH_DATA_FILE = Optional.empty();
@@ -93,7 +101,7 @@ public static File getEvidenceFile() {
             return EVIDENCE_FILE.orElse(null);
         }
         try {
-           EVIDENCE_FILE = Optional.of(getFilePath(EVIDENCE_FILE_NAME));
+           EVIDENCE_FILE = Optional.of(getFilePath(EVIDENCE_FILE_PATH));
            return EVIDENCE_FILE.get();
         } catch (Exception e) {
             EVIDENCE_FILE = Optional.empty();

device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionTests.java

@@ -52,7 +52,7 @@
 import java.util.concurrent.Executors;
 import java.util.concurrent.Future;
 
-import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_NAME;
+import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_PATH;
 import static fiftyone.pipeline.core.Constants.EVIDENCE_HTTPHEADER_PREFIX;
 import static fiftyone.pipeline.core.Constants.EVIDENCE_SEPERATOR;
 import static fiftyone.pipeline.engines.Constants.PerformanceProfiles.*;
@@ -99,7 +99,7 @@ public class DeviceDetectionTests {
     @BeforeClass
     public static void initClass() throws IOException {
         userAgents = new UserAgentGenerator(
-            FileFinder.getFilePath(UA_FILE_NAME));
+            FileFinder.getFilePath(UA_FILE_PATH));
     }
 
     private static String reportErrors(Collection<FlowError> errors) {

examples/device-detection/README.md

@@ -21,6 +21,14 @@ Among other things, the examples illustrate:
 - use of device detection pipeline for off-line processing tasks
 - configuring device detection trade-offs between speed and conserving memory
 
+## Data files
+
+The on-premise examples use the free Lite data file and the evidence files in
+`cxx/data/device-detection`, a folder of the
+[data](https://github.com/51Degrees/data) repository. An example that cannot
+find one of these files runs the data repository's download script for it,
+and logs the command it runs.
+
 ## Cloud resource keys
 
 You will require [resource keys](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-resource-keys)
@@ -114,7 +122,8 @@ PORT=8100 java -jar ./web/getting-started.onprem/target/device-detection-java-ex
 
 The on-premise web example takes its data file from the `51DEGREES_DD_PATH`
 environment variable, then from `TestDataFile`, and otherwise uses the free Lite file
-in `device-detection-data`.
+in `cxx/data/device-detection`, which it downloads before the server starts if it is
+missing.
 
 ### Native library access
 

examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedOnPrem.java

@@ -58,12 +58,13 @@
 public class GettingStartedOnPrem {
     private static final Logger logger = LoggerFactory.getLogger(GettingStartedOnPrem.class);
 
-    /* In this example, by default, the 51degrees "Lite" file needs to be somewhere in the project
-    space, or you may specify another file as a command line parameter.
+    /* In this example, by default, the 51degrees "Lite" file is used from the data repository in
+    cxx/data, and is downloaded if it is missing, or you may specify another file as a command line
+    parameter.
 
     Note that the Lite data file is only used for illustration, and has limited accuracy and
     capabilities. Find out about the Enterprise data file here: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-gettingstartedonprem.java&utm_term=lite_v_4_1_hash */
-    public static String LITE_V_4_1_HASH = "51Degrees-LiteV4.1.hash";
+    public static String LITE_V_4_1_HASH = "device-detection/51Degrees-LiteV4.1.hash";
     public static String ENTERPRISE_HASH = "TAC-HashV41.hash";
 
     public static void main(String[] args) throws Exception {
@@ -87,7 +88,8 @@ public static void run(String dataFile,
                            OutputStream outputStream) throws Exception {
         logger.info("Running GettingStarted example");
         try {
-            String dataFileLocation = getFilePath(dataFile).getAbsolutePath();
+            String dataFileLocation =
+                    DataFileHelper.findDataFile(dataFile).getAbsolutePath();
             // the location of the test data file is interpolated in the pipeline
             // configuration file
             System.setProperty("TestDataFile", dataFileLocation);

examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MatchMetrics.java

@@ -27,7 +27,6 @@
 import fiftyone.devicedetection.examples.shared.EvidenceHelper;
 import fiftyone.devicedetection.hash.engine.onpremise.data.DeviceDataHash;
 import fiftyone.devicedetection.hash.engine.onpremise.flowelements.DeviceDetectionHashEngine;
-import fiftyone.devicedetection.shared.testhelpers.FileUtils;
 import fiftyone.pipeline.core.data.ElementPropertyMetaData;
 import fiftyone.pipeline.core.data.FlowData;
 import fiftyone.pipeline.core.flowelements.Pipeline;
@@ -80,7 +79,7 @@ static void run(String dataFile, List<Map<String, String>> evidenceList,
                     boolean showDescs, OutputStream out) throws Exception {
         PrintWriter writer = new PrintWriter(out, true);
         if (Objects.isNull(dataFile)) {
-            dataFile = FileUtils.getHashFileName();
+            dataFile = DataFileHelper.getDataFileLocation(null);
         }
         logger.info("Constructing pipeline from file " + dataFile);
 

examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MetadataOnPrem.java

@@ -45,12 +45,13 @@
 public class MetadataOnPrem {
     private static final Logger logger = LoggerFactory.getLogger(GettingStartedOnPrem.class);
 
-    /* In this example, by default, the 51degrees "Lite" file needs to be somewhere in the project
-    space, or you may specify another file as a command line parameter.
+    /* In this example, by default, the 51degrees "Lite" file is used from the data repository in
+    cxx/data, and is downloaded if it is missing, or you may specify another file as a command line
+    parameter.
 
     Note that the Lite data file is only used for illustration, and has limited accuracy and
     capabilities. Find out about the Enterprise data file here: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-metadataonprem.java&utm_term=lite_v_4_1_hash */
-    public static String LITE_V_4_1_HASH = "51Degrees-LiteV4.1.hash";
+    public static String LITE_V_4_1_HASH = "device-detection/51Degrees-LiteV4.1.hash";
 
     public static void main(String[] args) throws Exception {
         configureLogback(getFilePath("logback.xml"));
@@ -63,7 +64,8 @@ public static void run(String dataFile, OutputStream output) throws Exception {
         logger.info("Running MetadataOnPrem example");
         String dataFileLocation;
         try {
-            dataFileLocation = getFilePath(dataFile).getAbsolutePath();
+            dataFileLocation =
+                    DataFileHelper.findDataFile(dataFile).getAbsolutePath();
         } catch (Exception e) {
             DataFileHelper.cantFindDataFile(dataFile);
             throw e;