diff --git a/AGENTS.md b/AGENTS.md index d163ccbfe..5a1d6f620 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -22,9 +22,18 @@ Tests and examples find data files with `FileFinder`, which searches below `project.root`. The surefire configuration sets that to `maven.multiModuleProjectDirectory`, which Maven takes from the nearest `.mvn` folder above the working directory. The only `.mvn` folder is at the root, so -the search covers `cxx/device-detection-data` and `cxx/ip-intelligence-data` +the search covers `cxx/data/device-detection` and `cxx/data/ip-intelligence` from wherever Maven is run. Do not add a `.mvn` folder below the root. +Those folders are the data repository, a submodule of cxx, where each free +data file is a `.link` until `get-data-files.ps1` or `get-data-files.sh` +downloads it. `ci/fetch-assets.ps1` runs the script, and the examples' +`DataFileHelper.findDataFile` runs it for a file that is missing. Look for a +free file by its folder and name, such as +`device-detection/51Degrees-LiteV4.1.hash`, so the search does not stop at a +file of the same name elsewhere, such as a Git LFS pointer in an old +`cxx/device-detection-data` folder. + ## Native builds Each on-premise engine compiles its own copy of the common C code from `cxx`, diff --git a/README.md b/README.md index 4ba31cf22..479f1ff36 100644 --- a/README.md +++ b/README.md @@ -44,7 +44,7 @@ Created from `pipeline-java`, `device-detection-java`, `ip-intelligence-java`, ` | `location` | The Location packages, `com.51degrees:geo-location*` | | `examples/device-detection` | Console and web examples for Device Detection | | `examples/ip-intelligence` | Console and web examples for IP Intelligence, including the mixed examples that run both products | -| `cxx` | Submodule of [51degrees-cxx](https://github.com/51Degrees/51degrees-cxx), with the C and C++ code the on-premise engines are built from and the data file submodules | +| `cxx` | Submodule of [51degrees-cxx](https://github.com/51Degrees/51degrees-cxx), with the C and C++ code the on-premise engines are built from and the [data](https://github.com/51Degrees/data) submodule in `cxx/data` | | `pom.xml` | The reactor that builds everything here. It is never published | | `ci` | The scripts the shared 51Degrees CI workflows call | @@ -55,10 +55,20 @@ git clone --recurse-submodules https://github.com/51Degrees/51degrees-java.git cd 51degrees-java ``` -The free Device Detection Lite data file in `cxx/device-detection-data` is -stored with [Git LFS](https://git-lfs.github.com/). Run `git lfs pull` in that -folder to fetch it. The IP Intelligence ASN file in `cxx/ip-intelligence-data` -is plain git. +The free data files come from the [data](https://github.com/51Degrees/data) +repository, which is the `cxx/data` submodule. Each one is listed there as a +`.link` file, and a script downloads them all into place: + +``` +cxx/data/get-data-files.sh +``` + +On Windows, run `cxx\data\get-data-files.ps1` in PowerShell instead. The +examples run the script themselves for a free file they need that is missing. + +In a clone made before the data repository was added, delete the +`cxx/device-detection-data` and `cxx/ip-intelligence-data` folders once cxx no +longer includes them, because Git leaves a removed submodule's folder on disk. ## Building @@ -89,8 +99,9 @@ mvn surefire:test ``` The cloud tests need a Resource Key, passed as `-DTestResourceKey=`. The -on-premise IP Intelligence tests need the Lite or Enterprise data file, which -is not in the repository. See `ip-intelligence/README.md` for how to get it. +on-premise tests need the free data files, which the script in +[Getting the code](#getting-the-code) downloads. For the Enterprise IP +Intelligence data file, see `ip-intelligence/README.md`. Tests find data files by searching from the root of this repository, so run Maven from here rather than from inside a package folder. diff --git a/ci/README.md b/ci/README.md index 1041809a8..a537b0271 100644 --- a/ci/README.md +++ b/ci/README.md @@ -21,8 +21,12 @@ repositories used to do for one package tree each. workflow instead tests the examples against the version it built, through [run-integration-tests.ps1](run-integration-tests.ps1). - [fetch-assets.ps1](fetch-assets.ps1) links every data file it downloads into - `cxx/device-detection-data` or `cxx/ip-intelligence-data`, where the tests - and examples find them. + `cxx/data/device-detection` or `cxx/data/ip-intelligence`, where the tests + and examples find them. Those are the folders of the + [data](https://github.com/51Degrees/data) repository, which is a submodule + of cxx. The script then runs the data repository's `get-data-files.ps1`, + which keeps the linked files and downloads the free files that are not + assets, which is the Asn file. - [run-performance-tests.ps1](run-performance-tests.ps1) measures the Pipeline API, Device Detection and IP Intelligence and writes one results file per configuration. The Device Detection and IP Intelligence metrics are prefixed diff --git a/ci/fetch-assets.ps1 b/ci/fetch-assets.ps1 index 23f509ea1..4062061b1 100644 --- a/ci/fetch-assets.ps1 +++ b/ci/fetch-assets.ps1 @@ -7,9 +7,11 @@ param ( $ErrorActionPreference = "Stop" # Tests and examples find data files by searching below the repository root, -# so every asset is linked into the data folders of the cxx submodule. -$deviceDetectionData = "$PSScriptRoot/../cxx/device-detection-data" -$ipIntelligenceData = "$PSScriptRoot/../cxx/ip-intelligence-data" +# so every asset is linked into the folders of the data repository, which is +# the data submodule of cxx. +$data = "$PSScriptRoot/../cxx/data" +$deviceDetectionData = "$data/device-detection" +$ipIntelligenceData = "$data/ip-intelligence" $deviceDetectionAssets = 'TAC-HashV41.hash', '51Degrees-LiteV4.1.hash', '20000 Evidence Records.yml', '20000 User Agents.csv' ./steps/fetch-assets.ps1 -DeviceDetection:$DeviceDetection -DeviceDetectionUrl:$DeviceDetectionUrl -Assets $deviceDetectionAssets @@ -24,6 +26,14 @@ New-Item -ItemType SymbolicLink -Force -Target "$PWD/assets/51Degrees-Enterprise # link it as that. New-Item -ItemType SymbolicLink -Force -Target "$PWD/assets/51Degrees-LiteIpiV41.ipi" -Path "$ipIntelligenceData/51Degrees-LiteV41.ipi" +# The data repository's script downloads every linked file that is not +# already there. The files linked above are kept, so this gets the free files +# that are not assets, which is the Asn file. +& "$data/get-data-files.ps1" +if (-not $?) { + throw "The data repository's get-data-files.ps1 could not download every file" +} + Write-Host "Assets hashes:" Get-FileHash -Algorithm MD5 -Path assets/* diff --git a/cxx b/cxx index 1bec7b570..e866f6d3c 160000 --- a/cxx +++ b/cxx @@ -1 +1 @@ -Subproject commit 1bec7b570575e2e180b52435599bef9b03e35ad8 +Subproject commit e866f6d3cea936587790f87967eff7863778b30e diff --git a/device-detection/README.md b/device-detection/README.md index c393a561e..73722fcb5 100644 --- a/device-detection/README.md +++ b/device-detection/README.md @@ -28,23 +28,24 @@ how to use this. #### On-Premise -If you are using on-premise detection, a "Lite" version of the data required is packaged -in this repository. It contains only a limited set of "essential" device detection properties. +If you are using on-premise detection, a free "Lite" version of the data required is +available from the [data](https://github.com/51Degrees/data) repository, which is the +`cxx/data` folder of this repository. It contains only a limited set of "essential" +device detection properties. You may want to license our complete data file containing all properties. [Details of our licenses](https://51degrees.com/pricing?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=on-premise) are available on our website. -If you want to use the lite file, you will need to install [GitLFS](https://git-lfs.github.com/), then: +To get the Lite file, run the data repository's download script from the root of +this repository: ``` -git lfs install +cxx/data/get-data-files.sh 51Degrees-LiteV4.1.hash ``` -Then, navigate to 'device-detection.hash.engine.on-premise/src/main/cxx/device-detection-cxx/device-detection-data' and execute: - -``` -git lfs pull -``` +On Windows, run `cxx\data\get-data-files.ps1 51Degrees-LiteV4.1.hash` in PowerShell +instead. The file is saved in `cxx/data/device-detection`, where the tests and +examples find it. ## Installation diff --git a/device-detection/device-detection.cloud/src/test/java/fiftyone/devicedetection/cloud/TestsBase.java b/device-detection/device-detection.cloud/src/test/java/fiftyone/devicedetection/cloud/TestsBase.java index ac8007cd1..44d233d6a 100644 --- a/device-detection/device-detection.cloud/src/test/java/fiftyone/devicedetection/cloud/TestsBase.java +++ b/device-detection/device-detection.cloud/src/test/java/fiftyone/devicedetection/cloud/TestsBase.java @@ -25,7 +25,7 @@ import fiftyone.devicedetection.shared.testhelpers.UserAgentGenerator; import fiftyone.pipeline.util.FileFinder; -import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_NAME; +import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_PATH; public class TestsBase { @@ -43,7 +43,7 @@ protected UserAgentGenerator getUserAgents() { protected void testInitialize() throws Exception { wrapper = new WrapperCloud(); userAgents = new UserAgentGenerator( - FileFinder.getFilePath(UA_FILE_NAME)); + FileFinder.getFilePath(UA_FILE_PATH)); } public void testCleanup() { diff --git a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/TestsBase.java b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/TestsBase.java index 0351da463..f3826ee0e 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/TestsBase.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/TestsBase.java @@ -26,7 +26,7 @@ import fiftyone.devicedetection.shared.testhelpers.FileUtils; import fiftyone.pipeline.engines.Constants; -import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_NAME; +import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_PATH; import static fiftyone.pipeline.util.FileFinder.getFilePath; public class TestsBase { @@ -47,7 +47,7 @@ protected void testInitialize(Constants.PerformanceProfiles profile) throws Exce FileUtils.getHashFile(), profile); userAgents = new UserAgentGenerator( - getFilePath(UA_FILE_NAME)); + getFilePath(UA_FILE_PATH)); } public void testCleanup() { diff --git a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/flowelements/EngineTests.java b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/flowelements/EngineTests.java index b73fbd83c..0b58bc098 100644 --- a/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/flowelements/EngineTests.java +++ b/device-detection/device-detection.hash.engine.on-premise/src/test/java/fiftyone/devicedetection/hash/engine/onpremise/flowelements/EngineTests.java @@ -33,7 +33,7 @@ import fiftyone.devicedetection.shared.testhelpers.FileUtils; import fiftyone.pipeline.engines.Constants; -import static fiftyone.devicedetection.shared.testhelpers.FileUtils.LITE_HASH_DATA_FILE_NAME; +import static fiftyone.devicedetection.shared.testhelpers.FileUtils.LITE_HASH_DATA_FILE_PATH; import static fiftyone.devicedetection.shared.testhelpers.FileUtils.TAC_HASH_DATA_FILE_NAME; import static fiftyone.pipeline.util.FileFinder.getFilePath; @@ -66,7 +66,7 @@ private void testDataSourceTier(String fileName) throws Exception { String tier = engine.getDataSourceTier(); engine.close(); - if (fileName.equals(LITE_HASH_DATA_FILE_NAME)) { + if (fileName.equals(LITE_HASH_DATA_FILE_PATH)) { assertEquals("Lite", tier); } else { @@ -81,7 +81,7 @@ private void testDataSourceTier(String fileName) throws Exception { */ @Test public void Engine_Hash_GetDataSourceTier_Lite() throws Exception { - testDataSourceTier(LITE_HASH_DATA_FILE_NAME); + testDataSourceTier(LITE_HASH_DATA_FILE_PATH); } /** diff --git a/device-detection/device-detection.shared/src/main/java/fiftyone/devicedetection/shared/testhelpers/FileUtils.java b/device-detection/device-detection.shared/src/main/java/fiftyone/devicedetection/shared/testhelpers/FileUtils.java index 8eff15a70..fe9e13a9e 100644 --- a/device-detection/device-detection.shared/src/main/java/fiftyone/devicedetection/shared/testhelpers/FileUtils.java +++ b/device-detection/device-detection.shared/src/main/java/fiftyone/devicedetection/shared/testhelpers/FileUtils.java @@ -47,6 +47,14 @@ public class FileUtils { private static File UA_FILE; public static final String EVIDENCE_FILE_NAME = "20000 Evidence Records.yml"; private static Optional EVIDENCE_FILE; + // Where each free file is in the data repository. A search for the path + // rather than the name does not stop at another file of the same name, + // such as a Git LFS pointer in an old device-detection-data folder. + public static final String LITE_HASH_DATA_FILE_PATH = + "device-detection/" + LITE_HASH_DATA_FILE_NAME; + public static final String UA_FILE_PATH = "device-detection/" + UA_FILE_NAME; + public static final String EVIDENCE_FILE_PATH = + "device-detection/" + EVIDENCE_FILE_NAME; /** * Helper to find the location of an Enterprise or Lite Hash file in the default search scope @@ -78,7 +86,7 @@ public static File getHashFile() { } catch (Exception ex) { try { // following throws exception if not found - HASH_DATA_FILE = Optional.of(getFilePath(LITE_HASH_DATA_FILE_NAME)); + HASH_DATA_FILE = Optional.of(getFilePath(LITE_HASH_DATA_FILE_PATH)); return HASH_DATA_FILE.get(); } catch (Exception exc) { HASH_DATA_FILE = Optional.empty(); @@ -93,7 +101,7 @@ public static File getEvidenceFile() { return EVIDENCE_FILE.orElse(null); } try { - EVIDENCE_FILE = Optional.of(getFilePath(EVIDENCE_FILE_NAME)); + EVIDENCE_FILE = Optional.of(getFilePath(EVIDENCE_FILE_PATH)); return EVIDENCE_FILE.get(); } catch (Exception e) { EVIDENCE_FILE = Optional.empty(); diff --git a/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionTests.java b/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionTests.java index 7d1bac24d..a59a0a6d6 100644 --- a/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionTests.java +++ b/device-detection/device-detection/src/test/java/fiftyone/devicedetection/DeviceDetectionTests.java @@ -52,7 +52,7 @@ import java.util.concurrent.Executors; import java.util.concurrent.Future; -import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_NAME; +import static fiftyone.devicedetection.shared.testhelpers.FileUtils.UA_FILE_PATH; import static fiftyone.pipeline.core.Constants.EVIDENCE_HTTPHEADER_PREFIX; import static fiftyone.pipeline.core.Constants.EVIDENCE_SEPERATOR; import static fiftyone.pipeline.engines.Constants.PerformanceProfiles.*; @@ -99,7 +99,7 @@ public class DeviceDetectionTests { @BeforeClass public static void initClass() throws IOException { userAgents = new UserAgentGenerator( - FileFinder.getFilePath(UA_FILE_NAME)); + FileFinder.getFilePath(UA_FILE_PATH)); } private static String reportErrors(Collection errors) { diff --git a/examples/device-detection/README.md b/examples/device-detection/README.md index 5d2edd339..1f42ecd77 100644 --- a/examples/device-detection/README.md +++ b/examples/device-detection/README.md @@ -21,6 +21,14 @@ Among other things, the examples illustrate: - use of device detection pipeline for off-line processing tasks - configuring device detection trade-offs between speed and conserving memory +## Data files + +The on-premise examples use the free Lite data file and the evidence files in +`cxx/data/device-detection`, a folder of the +[data](https://github.com/51Degrees/data) repository. An example that cannot +find one of these files runs the data repository's download script for it, +and logs the command it runs. + ## Cloud resource keys You will require [resource keys](https://51degrees.com/documentation/_info__resource_keys.html?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=cloud-resource-keys) @@ -114,7 +122,8 @@ PORT=8100 java -jar ./web/getting-started.onprem/target/device-detection-java-ex The on-premise web example takes its data file from the `51DEGREES_DD_PATH` environment variable, then from `TestDataFile`, and otherwise uses the free Lite file -in `device-detection-data`. +in `cxx/data/device-detection`, which it downloads before the server starts if it is +missing. ### Native library access diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedOnPrem.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedOnPrem.java index 9d8819565..332ef1dca 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedOnPrem.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/GettingStartedOnPrem.java @@ -58,12 +58,13 @@ public class GettingStartedOnPrem { private static final Logger logger = LoggerFactory.getLogger(GettingStartedOnPrem.class); - /* In this example, by default, the 51degrees "Lite" file needs to be somewhere in the project - space, or you may specify another file as a command line parameter. + /* In this example, by default, the 51degrees "Lite" file is used from the data repository in + cxx/data, and is downloaded if it is missing, or you may specify another file as a command line + parameter. Note that the Lite data file is only used for illustration, and has limited accuracy and capabilities. Find out about the Enterprise data file here: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-gettingstartedonprem.java&utm_term=lite_v_4_1_hash */ - public static String LITE_V_4_1_HASH = "51Degrees-LiteV4.1.hash"; + public static String LITE_V_4_1_HASH = "device-detection/51Degrees-LiteV4.1.hash"; public static String ENTERPRISE_HASH = "TAC-HashV41.hash"; public static void main(String[] args) throws Exception { @@ -87,7 +88,8 @@ public static void run(String dataFile, OutputStream outputStream) throws Exception { logger.info("Running GettingStarted example"); try { - String dataFileLocation = getFilePath(dataFile).getAbsolutePath(); + String dataFileLocation = + DataFileHelper.findDataFile(dataFile).getAbsolutePath(); // the location of the test data file is interpolated in the pipeline // configuration file System.setProperty("TestDataFile", dataFileLocation); diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MatchMetrics.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MatchMetrics.java index a54bfe71c..4c0aa55a1 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MatchMetrics.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MatchMetrics.java @@ -27,7 +27,6 @@ import fiftyone.devicedetection.examples.shared.EvidenceHelper; import fiftyone.devicedetection.hash.engine.onpremise.data.DeviceDataHash; import fiftyone.devicedetection.hash.engine.onpremise.flowelements.DeviceDetectionHashEngine; -import fiftyone.devicedetection.shared.testhelpers.FileUtils; import fiftyone.pipeline.core.data.ElementPropertyMetaData; import fiftyone.pipeline.core.data.FlowData; import fiftyone.pipeline.core.flowelements.Pipeline; @@ -80,7 +79,7 @@ static void run(String dataFile, List> evidenceList, boolean showDescs, OutputStream out) throws Exception { PrintWriter writer = new PrintWriter(out, true); if (Objects.isNull(dataFile)) { - dataFile = FileUtils.getHashFileName(); + dataFile = DataFileHelper.getDataFileLocation(null); } logger.info("Constructing pipeline from file " + dataFile); diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MetadataOnPrem.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MetadataOnPrem.java index a58e739ab..1dd9b1d21 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MetadataOnPrem.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/MetadataOnPrem.java @@ -45,12 +45,13 @@ public class MetadataOnPrem { private static final Logger logger = LoggerFactory.getLogger(GettingStartedOnPrem.class); - /* In this example, by default, the 51degrees "Lite" file needs to be somewhere in the project - space, or you may specify another file as a command line parameter. + /* In this example, by default, the 51degrees "Lite" file is used from the data repository in + cxx/data, and is downloaded if it is missing, or you may specify another file as a command line + parameter. Note that the Lite data file is only used for illustration, and has limited accuracy and capabilities. Find out about the Enterprise data file here: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-metadataonprem.java&utm_term=lite_v_4_1_hash */ - public static String LITE_V_4_1_HASH = "51Degrees-LiteV4.1.hash"; + public static String LITE_V_4_1_HASH = "device-detection/51Degrees-LiteV4.1.hash"; public static void main(String[] args) throws Exception { configureLogback(getFilePath("logback.xml")); @@ -63,7 +64,8 @@ public static void run(String dataFile, OutputStream output) throws Exception { logger.info("Running MetadataOnPrem example"); String dataFileLocation; try { - dataFileLocation = getFilePath(dataFile).getAbsolutePath(); + dataFileLocation = + DataFileHelper.findDataFile(dataFile).getAbsolutePath(); } catch (Exception e) { DataFileHelper.cantFindDataFile(dataFile); throw e; diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/OfflineProcessing.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/OfflineProcessing.java index 031c63368..8fc928a56 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/OfflineProcessing.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/OfflineProcessing.java @@ -67,22 +67,22 @@ */ public class OfflineProcessing { static final Logger logger = LoggerFactory.getLogger(OfflineProcessing.class); - // This 51degrees "Lite" file (distributed with the source) needs to - // be somewhere in the project space + // This 51degrees "Lite" file is used from the data repository in + // cxx/data, and is downloaded if it is missing // // Note that the Lite data file is only used for illustration, and has // limited accuracy and capabilities. Find out about the Enterprise data // file here: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-devicedetection-examples-console-offlineprocessing.java&utm_term=lite_v_4_1_hash public static final String LITE_V_4_1_HASH = - "device-detection-data/51Degrees-LiteV4.1.hash"; - // This 51degrees file of 20,000 examples (distributed with the source) - // needs to be somewhere in the project space + "device-detection/51Degrees-LiteV4.1.hash"; + // This 51degrees file of 20,000 examples is used from the same folder, + // and is downloaded if it is missing public static final String HEADER_EVIDENCE_YML = - "device-detection-data/20000 Evidence Records.yml"; + "device-detection/20000 Evidence Records.yml"; public static void main(String[] args) throws Exception { configureLogback(getFilePath("logback.xml")); - File evidenceFile = getFilePath(HEADER_EVIDENCE_YML); + File evidenceFile = DataFileHelper.findDataFile(HEADER_EVIDENCE_YML); run(LITE_V_4_1_HASH, Files.newInputStream(evidenceFile.toPath()), System.out); } @@ -100,7 +100,8 @@ public static void run(String dataFile, InputStream is, OutputStream os) throws String detectionFile; try { - detectionFile = getFilePath(dataFile).getAbsolutePath(); + detectionFile = + DataFileHelper.findDataFile(dataFile).getAbsolutePath(); } catch (Exception e) { DataFileHelper.cantFindDataFile(dataFile); throw e; diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/PerformanceBenchmark.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/PerformanceBenchmark.java index aeaf3180c..8c6f2b0fe 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/PerformanceBenchmark.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/PerformanceBenchmark.java @@ -388,8 +388,9 @@ public PerformanceConfiguration(Constants.PerformanceProfiles profile, * @example PerformanceBenchmark.java * The example illustrates a "clock-time" benchmark for assessing detection speed. * - * Using a YAML formatted evidence file - "20000 Evidence Records.yml" - supplied with the - * distribution or can be obtained from the [data repository on Github](https://github.com/51Degrees/device-detection-data/blob/master/20000%20Evidence%20Records.yml). + * Using a YAML formatted evidence file - "20000 Evidence Records.yml" - which is downloaded by + * the scripts in the [data repository on GitHub](https://github.com/51Degrees/data), and by the + * example itself when the file is missing. * * It's important to understand the trade-offs between performance, memory usage and accuracy, that * the 51Degrees pipeline configuration makes available, and this example shows a range of diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/ShareUsageExample.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/ShareUsageExample.java index 9e9ba2320..621ef0761 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/ShareUsageExample.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/ShareUsageExample.java @@ -22,6 +22,7 @@ package fiftyone.devicedetection.examples.console; +import fiftyone.devicedetection.examples.shared.DataFileHelper; import fiftyone.pipeline.core.data.FlowData; import fiftyone.pipeline.core.flowelements.Pipeline; import fiftyone.pipeline.core.flowelements.PipelineBuilder; @@ -75,9 +76,10 @@ public class ShareUsageExample { static final Logger logger = LoggerFactory.getLogger(ShareUsageExample.class); - // Evidence file with 20,000 examples + // Evidence file with 20,000 examples, from the data repository in + // cxx/data. The example downloads the file if it is missing. public static final String HEADER_EVIDENCE_YML = - "device-detection-data/20000 Evidence Records.yml"; + "device-detection/20000 Evidence Records.yml"; // Configurable settings private static final int MINIMUM_ENTRIES_PER_MESSAGE = 10; // Send after 10 entries (default is 50) @@ -100,7 +102,7 @@ public class ShareUsageExample { public static void main(String[] args) throws Exception { configureLogback(getFilePath("logback.xml")); - File evidenceFile = getFilePath(HEADER_EVIDENCE_YML); + File evidenceFile = DataFileHelper.findDataFile(HEADER_EVIDENCE_YML); run(Files.newInputStream(evidenceFile.toPath())); } diff --git a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/UpdateDataFile.java b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/UpdateDataFile.java index 6dae1a226..d3babcd27 100644 --- a/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/UpdateDataFile.java +++ b/examples/device-detection/console/src/main/java/fiftyone/devicedetection/examples/console/UpdateDataFile.java @@ -73,9 +73,9 @@ * ``` * ## Lite Data File * Lite data files (free-to-use, limited capabilities, no license key required) are created roughly - * once a month and cannot be updated using auto-update, they may be downloaded from - * [Github](https://github.com/51Degrees/device-detection-data) and are included with - * source distributions of this software. + * once a month and cannot be updated using auto-update, they may be downloaded with the + * scripts in the [data repository](https://github.com/51Degrees/data) on GitHub, which is + * the cxx/data folder of the source. * # Update on Start-Up * You can configure the pipeline builder to download an Enterprise data file on start-up. * ## Pre-Requisites diff --git a/examples/device-detection/console/src/main/resources/gettingStartedOnPrem.xml b/examples/device-detection/console/src/main/resources/gettingStartedOnPrem.xml index fed362bfd..ee634b786 100644 --- a/examples/device-detection/console/src/main/resources/gettingStartedOnPrem.xml +++ b/examples/device-detection/console/src/main/resources/gettingStartedOnPrem.xml @@ -5,7 +5,7 @@ false false - ${TestDataFile:-cxx/device-detection-data/51Degrees-LiteV4.1.hash} + ${TestDataFile:-cxx/data/device-detection/51Degrees-LiteV4.1.hash} LowMemory DeviceDetectionHashEngine diff --git a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/GettingStartedOnPremTest.java b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/GettingStartedOnPremTest.java index 1c272b52e..779a70fc2 100644 --- a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/GettingStartedOnPremTest.java +++ b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/GettingStartedOnPremTest.java @@ -31,7 +31,7 @@ public class GettingStartedOnPremTest { @Test public void gettingStartedOnPremTest() throws Exception { GettingStartedOnPrem.run(FileUtils.getHashFileName() == null - ? FileUtils.LITE_HASH_DATA_FILE_NAME + ? FileUtils.LITE_HASH_DATA_FILE_PATH : FileUtils.getHashFileName(), EvidenceHelper.setUpEvidence(), System.out); } diff --git a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/MetadataOnPremTest.java b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/MetadataOnPremTest.java index 32d3b8882..a128b1393 100644 --- a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/MetadataOnPremTest.java +++ b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/MetadataOnPremTest.java @@ -29,7 +29,7 @@ public class MetadataOnPremTest { @Test public void gettingMetaDataOnPremTest() throws Exception { MetadataOnPrem.run(FileUtils.getHashFileName() == null - ? FileUtils.LITE_HASH_DATA_FILE_NAME + ? FileUtils.LITE_HASH_DATA_FILE_PATH : FileUtils.getHashFileName(), System.out); } } \ No newline at end of file diff --git a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/OfflineProcessingTest.java b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/OfflineProcessingTest.java index 80d767c58..4861f14d6 100644 --- a/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/OfflineProcessingTest.java +++ b/examples/device-detection/console/src/test/java/fiftyone/devicedetection/examples/console/OfflineProcessingTest.java @@ -32,7 +32,7 @@ public class OfflineProcessingTest { @Test public void offlineProcessingTest() throws Exception { OfflineProcessing.run(FileUtils.getHashFileName() == null - ? FileUtils.LITE_HASH_DATA_FILE_NAME + ? FileUtils.LITE_HASH_DATA_FILE_PATH : FileUtils.getHashFileName(), new FileInputStream(Objects.requireNonNull(FileUtils.getEvidenceFile())), System.out); diff --git a/examples/device-detection/shared/src/main/java/fiftyone/devicedetection/examples/shared/DataFileHelper.java b/examples/device-detection/shared/src/main/java/fiftyone/devicedetection/examples/shared/DataFileHelper.java index c5aab3928..337f6e165 100644 --- a/examples/device-detection/shared/src/main/java/fiftyone/devicedetection/examples/shared/DataFileHelper.java +++ b/examples/device-detection/shared/src/main/java/fiftyone/devicedetection/examples/shared/DataFileHelper.java @@ -32,10 +32,15 @@ import org.slf4j.LoggerFactory; import java.io.File; +import java.io.IOException; import java.text.SimpleDateFormat; import java.time.Instant; import java.time.temporal.ChronoUnit; +import java.util.ArrayList; +import java.util.Arrays; import java.util.Date; +import java.util.List; +import java.util.Locale; import java.util.Objects; import static fiftyone.pipeline.util.FileFinder.getFilePath; @@ -43,6 +48,12 @@ public class DataFileHelper { static Logger logger = LoggerFactory.getLogger(DataFileHelper.class); + /** + * Sent with each download, so the Distributor can count the downloads + * made by these examples. + */ + private static final String DOWNLOAD_SOURCE = "examples-51degrees-java"; + public static class DatafileInfo { FiftyOneDataFile fileInfo; String tier; @@ -90,9 +101,10 @@ public static void logDataFileInfo(DeviceDetectionHashEngine engine) { "than {} days old. A more recent data file " + "may be needed to correctly detect the latest " + "devices, browsers, etc.", daysOld); - logger.info("The latest 'Lite' data file is available " + - "from the device-detection-data repository on GitHub " + - "https://github.com/51Degrees/device-detection-data. " + + logger.info("Run get-data-files.ps1 -Update or " + + "get-data-files.sh -u in cxx/data, which is the data " + + "repository https://github.com/51Degrees/data, to get " + + "the latest 'Lite' data file. " + "Find out about the Enterprise data file, which " + "includes automatic daily updates, on our pricing " + "page: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=shared-src-main-java-fiftyone-devicedetection-examples-shared-datafilehelper.java&utm_term=data-file-age-warning"); @@ -102,9 +114,10 @@ public static void logDataFileInfo(DeviceDetectionHashEngine engine) { public static void cantFindDataFile(String dataFile) { logger.error("Could not find the data file '{}' which must be " + "somewhere in the project space to be found.", dataFile); - logger.info("The latest 'Lite' data file is available " + - "from the device-detection-data repository on GitHub " + - "https://github.com/51Degrees/device-detection-data. " + + logger.info("The free 'Lite' data file and the evidence files " + + "are downloaded by get-data-files.ps1 or get-data-files.sh " + + "in cxx/data, which is the data repository " + + "https://github.com/51Degrees/data. " + "Find out about the Enterprise data file, which includes " + "automatic daily updates, on our pricing " + "page: https://51degrees.com/pricing?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=shared-src-main-java-fiftyone-devicedetection-examples-shared-datafilehelper.java&utm_term=data-file-not-found"); @@ -120,12 +133,12 @@ public static void cantFindDataFile(String dataFile) { @SuppressWarnings("RedundantThrows") public static File getEvidenceFile(String evidenceFilename) throws Exception { if (Objects.isNull(evidenceFilename)) { - evidenceFilename = FileUtils.EVIDENCE_FILE_NAME; + evidenceFilename = FileUtils.EVIDENCE_FILE_PATH; } File evidenceFile; try { - evidenceFile = FileFinder.getFilePath(evidenceFilename); + evidenceFile = findDataFile(evidenceFilename); } catch (Exception e) { logger.error("Could not find evidence file {}", evidenceFilename); throw e; @@ -134,7 +147,8 @@ public static File getEvidenceFile(String evidenceFilename) throws Exception { } /** - * Tries to find the passed file, or if null a default file + * Tries to find the passed file, or if null the first data file found, + * or the free Lite file when there is none * @param dataFilename a filename to find * @return a full pathname * @throws Exception if the file was not found @@ -144,13 +158,90 @@ public static String getDataFileLocation(String dataFilename) throws Exception { if (Objects.isNull(dataFilename)) { dataFilename = FileUtils.getHashFileName(); } + if (Objects.isNull(dataFilename)) { + dataFilename = FileUtils.LITE_HASH_DATA_FILE_PATH; + } String dataFileLocation; try { - dataFileLocation = getFilePath(dataFilename).getAbsolutePath(); + dataFileLocation = findDataFile(dataFilename).getAbsolutePath(); } catch (Exception e) { DataFileHelper.cantFindDataFile(dataFilename); throw e; } return dataFileLocation; } + + /** + * Finds a file below the project root, as + * {@link FileFinder#getFilePath(String)} does. A free data file that is + * missing has a {@code .link} in the data repository, so the data + * repository's script is run to download it, and the file is looked for + * again. + * @param file the name of the file, or the end of its path + * @return the file + * @throws IllegalArgumentException if the file was not found, naming the + * command that downloads it when there is one + */ + public static File findDataFile(String file) { + try { + return getFilePath(file); + } catch (IllegalArgumentException notFound) { + File link; + try { + link = getFilePath(file + ".link"); + } catch (IllegalArgumentException noLink) { + throw notFound; + } + String command = runDownload(link); + try { + return getFilePath(file); + } catch (IllegalArgumentException stillNotFound) { + String message = "Could not download '" + file + + "'. Run this command to download it: " + command; + logger.error(message); + throw new IllegalArgumentException(message, stillNotFound); + } + } + } + + /** + * Runs the data repository's script for the file that a link is for, + * and waits for it to finish. The data repository is the folder above + * the one the link is in. + * @param link the {@code .link} file + * @return the command, as it would be typed + */ + private static String runDownload(File link) { + String name = link.getName().substring( + 0, link.getName().length() - ".link".length()); + File data = link.getAbsoluteFile().getParentFile().getParentFile(); + List command; + if (System.getProperty("os.name", "") + .toLowerCase(Locale.ROOT).startsWith("windows")) { + command = Arrays.asList("powershell", "-NoProfile", + "-ExecutionPolicy", "Bypass", "-File", + new File(data, "get-data-files.ps1").getPath(), name, + "-Source", DOWNLOAD_SOURCE); + } else { + command = Arrays.asList("sh", + new File(data, "get-data-files.sh").getPath(), + "-s", DOWNLOAD_SOURCE, name); + } + List typed = new ArrayList<>(); + for (String argument : command) { + typed.add(argument.contains(" ") ? + "\"" + argument + "\"" : argument); + } + String line = String.join(" ", typed); + logger.info("'{}' is missing, so it is being downloaded by running {}", + name, line); + try { + new ProcessBuilder(command).inheritIO().start().waitFor(); + } catch (IOException e) { + logger.error("Could not run {}. {}", line, e.getMessage()); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return line; + } } diff --git a/examples/device-detection/web/getting-started.onprem/src/main/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPrem.java b/examples/device-detection/web/getting-started.onprem/src/main/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPrem.java index 22b31c746..8387720c6 100644 --- a/examples/device-detection/web/getting-started.onprem/src/main/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPrem.java +++ b/examples/device-detection/web/getting-started.onprem/src/main/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPrem.java @@ -22,6 +22,7 @@ package fiftyone.devicedetection.examples.web; +import fiftyone.devicedetection.examples.shared.DataFileHelper; import fiftyone.devicedetection.shared.DeviceData; import fiftyone.pipeline.core.configuration.PipelineOptions; import fiftyone.pipeline.core.configuration.PipelineOptionsFactory; @@ -34,6 +35,7 @@ import javax.servlet.http.HttpServlet; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; +import java.io.File; import java.io.PrintWriter; import static fiftyone.common.testhelpers.LogbackHelper.configureLogback; @@ -48,7 +50,8 @@ *

* The data file is taken from the 51DEGREES_DD_PATH environment variable, then * from TestDataFile (an environment variable or system property), and otherwise - * the free Lite file in device-detection-data is used. + * the free Lite file in cxx/data/device-detection is used, which is downloaded + * before the server starts if it is missing. *

* The server listens on port 8081 and stops when Enter is pressed. If the PORT * environment variable is set, the server listens on that port instead and runs @@ -63,6 +66,16 @@ public static void main(String[] args) throws Exception { configureLogback(getFilePath("logback.xml")); logger.info("Running Example {}", GettingStartedWebOnPrem.class); + // If the data file the configuration names is not at that path, it + // is searched for, and the free Lite file is downloaded if missing. + String dataFile = PipelineOptionsFactory.getOptionsFromFile( + getFilePath(resourceBase) + "/WEB-INF/51Degrees-OnPrem.xml") + .findAndSubstitute("DeviceDetectionHashEngine", "DataFile"); + if (!new File(dataFile).exists()) { + System.setProperty("TestDataFile", + DataFileHelper.getDataFileLocation(dataFile)); + } + String portEnv = System.getenv("PORT"); if (portEnv != null) { // Automated runs (e.g. the unified Selenium suite) inject a port and have no diff --git a/examples/device-detection/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml b/examples/device-detection/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml index f785a6607..bba6dd60e 100644 --- a/examples/device-detection/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml +++ b/examples/device-detection/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml @@ -32,7 +32,7 @@ variable, then from TestDataFile (an environment variable or system property), and otherwise the free Lite file is used. --> - ${51DEGREES_DD_PATH:-${TestDataFile:-cxx/device-detection-data/51Degrees-LiteV4.1.hash}} + ${51DEGREES_DD_PATH:-${TestDataFile:-cxx/data/device-detection/51Degrees-LiteV4.1.hash}} LowMemory DeviceDetectionHashEngine diff --git a/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPremTest.java b/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPremTest.java index 70636a724..d6a94896f 100644 --- a/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPremTest.java +++ b/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/GettingStartedWebOnPremTest.java @@ -43,7 +43,7 @@ public class GettingStartedWebOnPremTest { @BeforeClass public static void startJetty() throws Exception { System.setProperty("TestDataFile", FileUtils.getHashFile() == null - ? FileUtils.LITE_HASH_DATA_FILE_NAME + ? FileUtils.LITE_HASH_DATA_FILE_PATH : FileUtils.getHashFile().getAbsolutePath()); SERVER = EmbedJetty.startWebApp(getFilePath(resourceBase).getAbsolutePath(), 8081); } diff --git a/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/OnPremDataFileOptionTest.java b/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/OnPremDataFileOptionTest.java index 7c898e848..108fbd615 100644 --- a/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/OnPremDataFileOptionTest.java +++ b/examples/device-detection/web/getting-started.onprem/src/test/java/fiftyone/devicedetection/examples/web/OnPremDataFileOptionTest.java @@ -43,7 +43,7 @@ public class OnPremDataFileOptionTest { private static final String DD_PATH = "51DEGREES_DD_PATH"; private static final String TEST_DATA_FILE = "TestDataFile"; private static final String LITE = - "cxx/device-detection-data/51Degrees-LiteV4.1.hash"; + "cxx/data/device-detection/51Degrees-LiteV4.1.hash"; private String savedDdPath; private String savedTestDataFile; diff --git a/examples/ip-intelligence/.gitignore b/examples/ip-intelligence/.gitignore index 092783861..23e4fdfef 100644 --- a/examples/ip-intelligence/.gitignore +++ b/examples/ip-intelligence/.gitignore @@ -48,5 +48,3 @@ SESSIONS.ser 51Degrees-IPIV4EnterpriseIpiV41.ipi.gz 51Degrees-IPIV4AsnIpiV41.ipi 51Degrees-IPIV4AsnIpiV41.ipi.gz -# device detection data file used by the mixed examples -device-detection-data/ diff --git a/examples/ip-intelligence/README.md b/examples/ip-intelligence/README.md index 022af4b4a..3ebef2ce6 100644 --- a/examples/ip-intelligence/README.md +++ b/examples/ip-intelligence/README.md @@ -10,16 +10,20 @@ These examples are not distributed as maven jars and need to be built by you. ## Required Files -See [ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data/) -repository for instructions on obtaining the necessary data files for on-premise IP intelligence. - -**IMPORTANT**: The enterprise IP Intelligence data file must be placed in the `ip-intelligence-data` -directory at the root of this repository. The expected filename is `51Degrees-EnterpriseIpiV41.ipi`. - -Two free data files are also usable for many of the examples. The ASN file -(`51Degrees-IPIV4AsnIpiV41.ipi`) is included in the `ip-intelligence-data` submodule, and the -Lite file (`51Degrees-LiteV41.ipi`) can be downloaded by running -`ip-intelligence-data/get-lite-file-from-azure.ps1` (or `.sh`). When the enterprise file is +The data files for on-premise IP intelligence are kept in `cxx/data/ip-intelligence`, a +folder of the [data](https://github.com/51Degrees/data) repository, which is a submodule of +cxx. + +**IMPORTANT**: The enterprise IP Intelligence data file must be placed in the +`cxx/data/ip-intelligence` directory of this repository. The expected filename is +`51Degrees-EnterpriseIpiV41.ipi`. + +Two free data files are also usable for many of the examples, the ASN file +(`51Degrees-IPIV4AsnIpiV41.ipi`) and the Lite file (`51Degrees-LiteV41.ipi`). Run +`cxx/data/get-data-files.sh ip-intelligence` from the root of this repository to download +them, or `cxx\data\get-data-files.ps1 ip-intelligence` in PowerShell on Windows. An example +that needs one of them and cannot find it runs the script for that file, and logs the +command it runs. When the enterprise file is absent, the tests for examples that work with any data file fall back to the ASN file and then the Lite file. The OfflineProcessing and PerformanceBenchmark examples need the RegisteredName property, which the ASN file does not contain, so their tests fall back to @@ -27,12 +31,16 @@ the Lite file only. Tests for examples needing enterprise-only properties (Metri and Suspicious) are skipped instead. ``` -ip-intelligence-java-examples/ -├── ip-intelligence-data/ -│ └── 51Degrees-EnterpriseIpiV41.ipi -├── console/ -├── web/ -└── shared/ +51degrees-java/ +├── cxx/ +│ └── data/ +│ └── ip-intelligence/ +│ └── 51Degrees-EnterpriseIpiV41.ipi +└── examples/ + └── ip-intelligence/ + ├── console/ + ├── web/ + └── shared/ ``` The examples locate the data file in the following order: @@ -40,9 +48,9 @@ The examples locate the data file in the following order: 1. The `51DEGREES_IPI_PATH` environment variable or system property, which can be set to an explicit path to the data file. 2. A search of the project folder hierarchy for the expected file name. -3. The expected location `ip-intelligence-data/51Degrees-EnterpriseIpiV41.ipi` +3. The expected location `cxx/data/ip-intelligence/51Degrees-EnterpriseIpiV41.ipi` (the configuration files for the getting started examples fall back to the - free Lite file `ip-intelligence-data/51Degrees-LiteV41.ipi`). + free Lite file `cxx/data/ip-intelligence/51Degrees-LiteV41.ipi`). This project contains sub-modules - **console**, giving examples that are intended to be run from the command line/console and **web**, illustrating use diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/CompareOnPrem.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/CompareOnPrem.java index 06538b059..b8df87518 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/CompareOnPrem.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/CompareOnPrem.java @@ -112,10 +112,10 @@ public class CompareOnPrem { /** * Relative path to the CSV file of truth records distributed with the - * ip-intelligence-data submodule. + * data repository, in cxx/data/ip-intelligence. */ public static final String GEOIP_COMPARISON_EVIDENCE_REL_PATH = - "ip-intelligence-data/geoip_comparison_evidence.csv"; + "ip-intelligence/geoip_comparison_evidence.csv"; /** * Default name of the output file written to the working directory. diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedMixed.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedMixed.java index 9bab40ab5..07f33c130 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedMixed.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/GettingStartedMixed.java @@ -83,7 +83,7 @@ public class GettingStartedMixed { private static final Logger logger = LoggerFactory.getLogger(GettingStartedMixed.class); - public static final String DD_DATA_FILE_DEFAULT = "device-detection-data/51Degrees-LiteV4.1.hash"; + public static final String DD_DATA_FILE_DEFAULT = "device-detection/51Degrees-LiteV4.1.hash"; public static final String IPI_DATA_FILE_DEFAULT = DataFileHelper.ENTERPRISE_DATA_FILE_REL_PATH; public static void main(String[] args) throws Exception { @@ -271,11 +271,12 @@ private static String resolveDataFile(String dataFile) throws IOException { if (Files.exists(relativePath)) { return relativePath.toAbsolutePath().toString(); } - // Try FileFinder (searches within project directory) + // Search within the project directory, downloading the free Lite + // file if it is missing try { - return getFilePath(dataFile).getAbsolutePath(); + return DataFileHelper.findDataFile(dataFile).getAbsolutePath(); } catch (Exception e) { - throw new IOException("Data file not found: " + dataFile); + throw new IOException("Data file not found: " + dataFile, e); } } } diff --git a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/OfflineProcessing.java b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/OfflineProcessing.java index 983d25997..608989b73 100644 --- a/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/OfflineProcessing.java +++ b/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/OfflineProcessing.java @@ -40,7 +40,7 @@ * advanced operation this example can be used to experiment with tuning IP Intelligence for * performance using Performance Profile settings. * - * Evidence files can be obtained from the [ip-intelligence-data repository](https://github.com/51Degrees/ip-intelligence-data). + * Evidence files can be obtained from the [data repository](https://github.com/51Degrees/data). * * This example is available in full on [GitHub](https://github.com/51Degrees/51degrees-java/blob/main/examples/ip-intelligence/console/src/main/java/fiftyone/ipintelligence/examples/console/OfflineProcessing.java). * @@ -99,10 +99,10 @@ public class OfflineProcessing { // be somewhere in the project space // // For testing, contact us to obtain an enterprise data file: https://51degrees.com/contact-us?utm_source=code&utm_medium=example&utm_campaign=51degrees-java&utm_content=console-src-main-java-fiftyone-ipintelligence-examples-console-offlineprocessing.java&utm_term=datadir - private static final String dataDir = "ip-intelligence-data"; + private static final String dataDir = "ip-intelligence"; // This 51degrees file of 20,000 examples (distributed with the source) // needs to be somewhere in the project space. Additional evidence files - // can be obtained from the [ip-intelligence-data repository](https://github.com/51Degrees/ip-intelligence-data) + // can be obtained from the [data repository](https://github.com/51Degrees/data) public static final String HEADER_EVIDENCE_YML = dataDir + "/evidence.yml"; diff --git a/examples/ip-intelligence/console/src/main/resources/gettingStartedMixed.xml b/examples/ip-intelligence/console/src/main/resources/gettingStartedMixed.xml index c6dba02ab..cfb3acbcf 100644 --- a/examples/ip-intelligence/console/src/main/resources/gettingStartedMixed.xml +++ b/examples/ip-intelligence/console/src/main/resources/gettingStartedMixed.xml @@ -5,7 +5,7 @@ false false - ${TestDeviceDetectionDataFile:-cxx/device-detection-data/51Degrees-LiteV4.1.hash} + ${TestDeviceDetectionDataFile:-cxx/data/device-detection/51Degrees-LiteV4.1.hash} LowMemory DeviceDetectionHashEngine @@ -14,7 +14,7 @@ false false - ${TestDataFile:-cxx/ip-intelligence-data/51Degrees-LiteV41.ipi} + ${TestDataFile:-cxx/data/ip-intelligence/51Degrees-LiteV41.ipi} LowMemory IPIntelligenceOnPremiseEngine diff --git a/examples/ip-intelligence/console/src/main/resources/gettingStartedOnPrem.xml b/examples/ip-intelligence/console/src/main/resources/gettingStartedOnPrem.xml index 7f1171389..f156a40b5 100644 --- a/examples/ip-intelligence/console/src/main/resources/gettingStartedOnPrem.xml +++ b/examples/ip-intelligence/console/src/main/resources/gettingStartedOnPrem.xml @@ -6,7 +6,7 @@ false false - ${TestDataFile:-cxx/ip-intelligence-data/51Degrees-LiteV41.ipi} + ${TestDataFile:-cxx/data/ip-intelligence/51Degrees-LiteV41.ipi} LowMemory diff --git a/examples/ip-intelligence/shared/src/main/java/fiftyone/ipintelligence/examples/shared/DataFileHelper.java b/examples/ip-intelligence/shared/src/main/java/fiftyone/ipintelligence/examples/shared/DataFileHelper.java index d448834d4..4cfa22723 100644 --- a/examples/ip-intelligence/shared/src/main/java/fiftyone/ipintelligence/examples/shared/DataFileHelper.java +++ b/examples/ip-intelligence/shared/src/main/java/fiftyone/ipintelligence/examples/shared/DataFileHelper.java @@ -39,7 +39,11 @@ import java.text.SimpleDateFormat; import java.time.Instant; import java.time.temporal.ChronoUnit; +import java.util.ArrayList; +import java.util.Arrays; import java.util.Date; +import java.util.List; +import java.util.Locale; import java.util.Objects; import static fiftyone.pipeline.util.FileFinder.getFilePath; @@ -48,31 +52,38 @@ public class DataFileHelper { static Logger logger = LoggerFactory.getLogger(DataFileHelper.class); /** - * Relative path to the enterprise IP Intelligence data file from the examples repository root. + * Sent with each download, so the Distributor can count the downloads + * made by these examples. + */ + private static final String DOWNLOAD_SOURCE = "examples-51degrees-java"; + + /** + * Folder and name of the enterprise IP Intelligence data file, which + * belongs in cxx/data/ip-intelligence, a folder of the data repository. * This path is used by both console and web examples for testing. - * The data file is downloaded to this location by CI scripts. + * The data file is linked to this location by CI scripts. */ - public static final String ENTERPRISE_DATA_FILE_REL_PATH = "ip-intelligence-data/51Degrees-EnterpriseIpiV41.ipi"; + public static final String ENTERPRISE_DATA_FILE_REL_PATH = "ip-intelligence/51Degrees-EnterpriseIpiV41.ipi"; /** - * Relative path to the freely downloadable Lite IP Intelligence data file. - * Run ip-intelligence-data/get-lite-file-from-azure.ps1 (or .sh) to obtain it. + * Folder and name of the free Lite IP Intelligence data file, which the + * data repository's get-data-files script downloads to that folder. * It contains the RegisteredCountry, RegisteredName and RegisteredOwner properties. */ - public static final String LITE_DATA_FILE_REL_PATH = "ip-intelligence-data/51Degrees-LiteV41.ipi"; + public static final String LITE_DATA_FILE_REL_PATH = "ip-intelligence/51Degrees-LiteV41.ipi"; /** - * Relative path to the ASN IP Intelligence data file which is part of the - * ip-intelligence-data submodule. It contains the Asn and AsnName properties. + * Folder and name of the free ASN IP Intelligence data file, which the + * same script downloads. It contains the Asn and AsnName properties. */ - public static final String ASN_DATA_FILE_REL_PATH = "ip-intelligence-data/51Degrees-IPIV4AsnIpiV41.ipi"; + public static final String ASN_DATA_FILE_REL_PATH = "ip-intelligence/51Degrees-IPIV4AsnIpiV41.ipi"; /** * Find the best IP Intelligence data file available, preferring the - * enterprise file, then the ASN file from the ip-intelligence-data - * submodule, then the Lite file. Suitable for examples that work with - * whatever properties the data file contains, where any data file - * allows the example to run. + * enterprise file, then the free ASN file, then the free Lite file, + * and downloading a free file that is missing. Suitable for examples + * that work with whatever properties the data file contains, where any + * data file allows the example to run. * @return a resolved data file path, or null when none is available */ public static String findAvailableDataFile() { @@ -180,7 +191,7 @@ public static File getEvidenceFile(String evidenceFilename) throws Exception { File evidenceFile; try { - evidenceFile = FileFinder.getFilePath(evidenceFilename); + evidenceFile = findDataFile(evidenceFilename); } catch (Exception e) { logger.error("Could not find evidence file {}", evidenceFilename); throw e; @@ -244,13 +255,92 @@ public static String getDataFileLocation(String dataFilename) throws Exception { if (Files.exists(dataPath)) { return dataPath.toAbsolutePath().toString(); } - DataFileHelper.cantFindDataFile(dataFilename); - throw new IOException("Data file not found at relative path: " + dataFilename + - ". Searched in project directories and current working directory." + - " An explicit path can be supplied via the " + IPI_PATH_ENV_VAR + - " environment variable."); + // Download it, if it is a free data file + try { + return findDataFile(dataFilename).getAbsolutePath(); + } catch (IllegalArgumentException notFound) { + DataFileHelper.cantFindDataFile(dataFilename); + throw new IOException("Data file not found at relative path: " + dataFilename + + ". Searched in project directories and current working directory." + + " An explicit path can be supplied via the " + IPI_PATH_ENV_VAR + + " environment variable.", notFound); + } } } } } + + /** + * Finds a file below the project root, as + * {@link FileFinder#getFilePath(String)} does. A free data file that is + * missing has a {@code .link} in the data repository, so the data + * repository's script is run to download it, and the file is looked for + * again. + * @param file the name of the file, or the end of its path + * @return the file + * @throws IllegalArgumentException if the file was not found, naming the + * command that downloads it when there is one + */ + public static File findDataFile(String file) { + try { + return getFilePath(file); + } catch (IllegalArgumentException notFound) { + File link; + try { + link = getFilePath(file + ".link"); + } catch (IllegalArgumentException noLink) { + throw notFound; + } + String command = runDownload(link); + try { + return getFilePath(file); + } catch (IllegalArgumentException stillNotFound) { + String message = "Could not download '" + file + + "'. Run this command to download it: " + command; + logger.error(message); + throw new IllegalArgumentException(message, stillNotFound); + } + } + } + + /** + * Runs the data repository's script for the file that a link is for, + * and waits for it to finish. The data repository is the folder above + * the one the link is in. + * @param link the {@code .link} file + * @return the command, as it would be typed + */ + private static String runDownload(File link) { + String name = link.getName().substring( + 0, link.getName().length() - ".link".length()); + File data = link.getAbsoluteFile().getParentFile().getParentFile(); + List command; + if (System.getProperty("os.name", "") + .toLowerCase(Locale.ROOT).startsWith("windows")) { + command = Arrays.asList("powershell", "-NoProfile", + "-ExecutionPolicy", "Bypass", "-File", + new File(data, "get-data-files.ps1").getPath(), name, + "-Source", DOWNLOAD_SOURCE); + } else { + command = Arrays.asList("sh", + new File(data, "get-data-files.sh").getPath(), + "-s", DOWNLOAD_SOURCE, name); + } + List typed = new ArrayList<>(); + for (String argument : command) { + typed.add(argument.contains(" ") ? + "\"" + argument + "\"" : argument); + } + String line = String.join(" ", typed); + logger.info("'{}' is missing, so it is being downloaded by running {}", + name, line); + try { + new ProcessBuilder(command).inheritIO().start().waitFor(); + } catch (IOException e) { + logger.error("Could not run {}. {}", line, e.getMessage()); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + return line; + } } diff --git a/examples/ip-intelligence/web/getting-started.api/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedApi.java b/examples/ip-intelligence/web/getting-started.api/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedApi.java index 2de381969..6e3fa2e8a 100644 --- a/examples/ip-intelligence/web/getting-started.api/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedApi.java +++ b/examples/ip-intelligence/web/getting-started.api/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedApi.java @@ -94,7 +94,7 @@ * and a JSON builder element which serialises the results. *

* The data file can be supplied as a command line argument. When no argument is given - * the enterprise data file is looked for in the ip-intelligence-data directory. + * the enterprise data file is looked for in the cxx/data/ip-intelligence directory. *

* Note that the .NET version of this example also includes a device detection engine * in the same pipeline. This Java version is IP Intelligence only. diff --git a/examples/ip-intelligence/web/getting-started.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebMixed.java b/examples/ip-intelligence/web/getting-started.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebMixed.java index 9969e3262..5b7331767 100644 --- a/examples/ip-intelligence/web/getting-started.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebMixed.java +++ b/examples/ip-intelligence/web/getting-started.mixed/src/main/java/fiftyone/ipintelligence/examples/web/GettingStartedWebMixed.java @@ -87,7 +87,7 @@ public class GettingStartedWebMixed extends HttpServlet { private static final long serialVersionUID = 1734154705981153541L; public static Logger logger = LoggerFactory.getLogger(GettingStartedWebMixed.class); - public static final String DD_DATA_FILE_DEFAULT = "device-detection-data/51Degrees-LiteV4.1.hash"; + public static final String DD_DATA_FILE_DEFAULT = "device-detection/51Degrees-LiteV4.1.hash"; public static void main(String[] args) throws Exception { configureLogback(getFilePath("logback.xml")); @@ -310,10 +310,13 @@ private static String resolveDataFile(String dataFile) throws IOException { if (Files.exists(relativePath)) { return relativePath.toAbsolutePath().toString(); } + // Search within the project directory, downloading the free Lite + // file if it is missing try { - return getFilePath(dataFile).getAbsolutePath(); + return fiftyone.ipintelligence.examples.shared.DataFileHelper + .findDataFile(dataFile).getAbsolutePath(); } catch (Exception e) { - throw new IOException("Data file not found: " + dataFile); + throw new IOException("Data file not found: " + dataFile, e); } } } diff --git a/examples/ip-intelligence/web/getting-started.mixed/src/main/webapp/WEB-INF/51Degrees-Mixed.xml b/examples/ip-intelligence/web/getting-started.mixed/src/main/webapp/WEB-INF/51Degrees-Mixed.xml index 95f859502..127b4665d 100644 --- a/examples/ip-intelligence/web/getting-started.mixed/src/main/webapp/WEB-INF/51Degrees-Mixed.xml +++ b/examples/ip-intelligence/web/getting-started.mixed/src/main/webapp/WEB-INF/51Degrees-Mixed.xml @@ -27,7 +27,7 @@ false false - ${TestDeviceDetectionDataFile:-cxx/device-detection-data/51Degrees-LiteV4.1.hash} + ${TestDeviceDetectionDataFile:-cxx/data/device-detection/51Degrees-LiteV4.1.hash} LowMemory DeviceDetectionHashEngine @@ -36,7 +36,7 @@ false false - ${TestDataFile:-cxx/ip-intelligence-data/51Degrees-EnterpriseIpiV41.ipi} + ${TestDataFile:-cxx/data/ip-intelligence/51Degrees-EnterpriseIpiV41.ipi} LowMemory IPIntelligenceOnPremiseEngine diff --git a/examples/ip-intelligence/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml b/examples/ip-intelligence/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml index d1bc62de7..c8e8d515b 100644 --- a/examples/ip-intelligence/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml +++ b/examples/ip-intelligence/web/getting-started.onprem/src/main/webapp/WEB-INF/51Degrees-OnPrem.xml @@ -29,7 +29,7 @@ false - ${TestDataFile:-cxx/ip-intelligence-data/51Degrees-EnterpriseIpiV41.ipi} + ${TestDataFile:-cxx/data/ip-intelligence/51Degrees-EnterpriseIpiV41.ipi} LowMemory IPIntelligenceOnPremiseEngine diff --git a/examples/ip-intelligence/web/shared/src/main/java/fiftyone/ipintelligence/examples/web/HtmlContentHelper.java b/examples/ip-intelligence/web/shared/src/main/java/fiftyone/ipintelligence/examples/web/HtmlContentHelper.java index 2425e1be5..0d70be0f1 100644 --- a/examples/ip-intelligence/web/shared/src/main/java/fiftyone/ipintelligence/examples/web/HtmlContentHelper.java +++ b/examples/ip-intelligence/web/shared/src/main/java/fiftyone/ipintelligence/examples/web/HtmlContentHelper.java @@ -78,8 +78,8 @@ public static String getDataFileAgeWarning(FlowData flowData) { return "

\n" + " The IP intelligence data file is " + daysOld + " days old. " + "A more recent data file may be needed for the most accurate IP " + - "intelligence results. The latest Lite data file is available from the " + - "ip-intelligence-data " + + "intelligence results. The latest Lite data file is downloaded by the " + + "scripts in the data " + "repository on GitHub. Find out about the full Enterprise data file, " + "which includes automatic daily updates, on our " + "pricing page.\n" + diff --git a/ip-intelligence/README.md b/ip-intelligence/README.md index 2b087fe20..3d194b87b 100644 --- a/ip-intelligence/README.md +++ b/ip-intelligence/README.md @@ -33,14 +33,13 @@ how to use this. In order to perform IP intelligence on-premise, you will need to use a data file. -[ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data/) repository instructs how to obtain a 'Lite' data file, otherwise [contact us](https://51degrees.com/contact-us?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=on-premise) to obtain an 'Enterprise' data file. +The free 'Lite' and Asn data files come from the [data](https://github.com/51Degrees/data) repository, which is the `cxx/data` folder of this repository. Run `cxx/data/get-data-files.sh ip-intelligence` from the root of this repository to download them, or `cxx\data\get-data-files.ps1 ip-intelligence` in PowerShell on Windows. Otherwise [contact us](https://51degrees.com/contact-us?utm_source=github&utm_medium=readme&utm_campaign=51degrees-java&utm_content=readme.md&utm_term=on-premise) to obtain an 'Enterprise' data file. For tests, the data file can be supplied explicitly by setting the `51DEGREES_IPI_PATH` environment variable or system property to the path of the .ipi file. When it is not set, the project space is searched for the -expected file name, with the free 'Lite' file expected in the -[ip-intelligence-data](https://github.com/51Degrees/ip-intelligence-data/) -location. +expected file name, with the free 'Lite' file expected in +`cxx/data/ip-intelligence`. ## Installation diff --git a/ip-intelligence/ip-intelligence.translation/pom.xml b/ip-intelligence/ip-intelligence.translation/pom.xml index 83cbc8865..a620ab341 100644 --- a/ip-intelligence/ip-intelligence.translation/pom.xml +++ b/ip-intelligence/ip-intelligence.translation/pom.xml @@ -60,9 +60,11 @@ + they are copied in at build time from the data repository in + cxx/data/ip-intelligence/Translations, as the .NET project + embeds the same files. A folder that is not there is skipped + without an error, so TranslationTests, which read the files + from the classpath, are what show they were copied. --> org.apache.maven.plugins maven-resources-plugin @@ -78,13 +80,13 @@ ${project.build.outputDirectory}/fiftyone/ipintelligence/translation - ${project.basedir}/../../cxx/ip-intelligence-data/Translations + ${project.basedir}/../../cxx/data/ip-intelligence/Translations countrycodes.en_GB.yml - ${project.basedir}/../../cxx/ip-intelligence-data/Translations/OSM + ${project.basedir}/../../cxx/data/ip-intelligence/Translations/OSM countries.*.yml